Repository: apache/beam
Pull request #34539: consume flow control settings updates from job settings
Author: login: m-trieu
Labels: name: runners
name: dataflow
name: Next Action: Reviewers
Created: 2025-04-03T22:37:38Z | Merged: 2025-04-16T14:00:11Z

consume flow control settings updates from job settings, lazily create streams/channels to use new flow control settings
- propogate current flow control settings to channel creation
- consume new flow control settings in `ChannelCache`
- lazily recreate streams in `WindmillConnection`  to pick up new flow control settings
- update tests


------------------------

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

 - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
 - [ ] Update `CHANGES.md` with noteworthy changes.
 - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).

To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)

GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)

See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.



Diff:
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
index 30782735d777..daca54fe1587 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
@@ -17,7 +17,7 @@
  */
 package org.apache.beam.runners.dataflow.worker;
 
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.remoteChannel;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.remoteChannel;
 
 import com.google.api.services.dataflow.model.MapTask;
 import com.google.auto.value.AutoValue;
@@ -87,7 +87,6 @@
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.GrpcWindmillStreamFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCache;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCachingRemoteStubFactory;
-import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCachingStubFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.IsolationChannel;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillStubFactoryFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillStubFactoryFactoryImpl;
@@ -243,10 +242,11 @@ private StreamingDataflowWorker(
     @Nullable ChannelzServlet channelzServlet = null;
     Consumer<PrintWriter> getDataStatusProvider;
     Supplier<Long> currentActiveCommitBytesProvider;
-
+    ChannelCache channelCache = null;
     if (options.isEnableStreamingEngine() && options.getIsWindmillServiceDirectPathEnabled()) {
       // Direct path pipelines.
       WeightedSemaphore<Commit> maxCommitByteSemaphore = Commits.maxCommitByteSemaphore();
+      channelCache = createChannelCache(options, configFetcher);
       FanOutStreamingEngineWorkerHarness fanOutStreamingEngineWorkerHarness =
           FanOutStreamingEngineWorkerHarness.create(
               createJobHeader(options, clientId),
@@ -273,7 +273,7 @@ private StreamingDataflowWorker(
                                 processingContext,
                                 getWorkStreamLatencies);
                           }),
-              createFanOutStubFactory(options),
+              ChannelCachingRemoteStubFactory.create(options.getGcpCredential(), channelCache),
               GetWorkBudgetDistributors.distributeEvenly(),
               Preconditions.checkNotNull(dispatcherClient),
               commitWorkStream ->
@@ -384,6 +384,7 @@ private StreamingDataflowWorker(
             .setChannelzServlet(channelzServlet)
             .setGetDataStatusProvider(getDataStatusProvider)
             .setCurrentActiveCommitBytes(currentActiveCommitBytesProvider)
+            .setChannelCache(channelCache)
             .build();
 
     LOG.debug("isDirectPathEnabled: {}", options.getIsWindmillServiceDirectPathEnabled());
@@ -620,19 +621,28 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
         StreamingDataflowWorker.class.getSimpleName());
   }
 
-  private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+  private static ChannelCache createChannelCache(
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            (currentFlowControlSettings, serviceAddress) -> {
+              // IsolationChannel will create and manage separate RPC channels to the same
+              // serviceAddress.
+              return IsolationChannel.create(
+                  () ->
+                      remoteChannel(
+                          serviceAddress,
+                          workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec(),
+                          currentFlowControlSettings),
+                  currentFlowControlSettings.getOnReadyThresholdBytes());
+            });
+    configFetcher
+        .getGlobalConfigHandle()
+        .registerConfigObserver(
+            config ->
+                channelCache.consumeFlowControlSettings(
+                    config.userWorkerJobSettings().getFlowControlSettings()));
+    return channelCache;
   }
 
   @VisibleForTesting
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandle.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandle.java
index 6f75ba887473..78810b35c889 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandle.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandle.java
@@ -18,7 +18,6 @@
 package org.apache.beam.runners.dataflow.worker.streaming.config;
 
 import java.util.function.Consumer;
-import javax.annotation.Nonnull;
 import javax.annotation.concurrent.ThreadSafe;
 import org.apache.beam.sdk.annotations.Internal;
 
@@ -33,5 +32,5 @@ public interface StreamingGlobalConfigHandle {
    * Subscribe to config updates by registering a callback. Callback should be called the first time
    * with settings, if any. The callback could execute inline before the method returns.
    */
-  void registerConfigObserver(@Nonnull Consumer<StreamingGlobalConfig> callback);
+  void registerConfigObserver(Consumer<StreamingGlobalConfig> callback);
 }
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/StreamingWorkerStatusPages.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/StreamingWorkerStatusPages.java
index ddfc6809231a..4be367e6cb72 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/StreamingWorkerStatusPages.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/StreamingWorkerStatusPages.java
@@ -44,6 +44,7 @@
 import org.apache.beam.runners.dataflow.worker.util.BoundedQueueExecutor;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.ChannelzServlet;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.GrpcWindmillStreamFactory;
+import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCache;
 import org.apache.beam.runners.dataflow.worker.windmill.state.WindmillStateCache;
 import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
@@ -79,6 +80,7 @@ public final class StreamingWorkerStatusPages {
   private final @Nullable GrpcWindmillStreamFactory windmillStreamFactory;
   private final DebugCapture.@Nullable Manager debugCapture;
   private final @Nullable ChannelzServlet channelzServlet;
+  private final @Nullable ChannelCache channelCache;
 
   private final AtomicReference<StreamingGlobalConfig> globalConfig = new AtomicReference<>();
 
@@ -96,7 +98,8 @@ public final class StreamingWorkerStatusPages {
       Consumer<PrintWriter> getDataStatusProvider,
       BoundedQueueExecutor workUnitExecutor,
       ScheduledExecutorService statusPageDumper,
-      StreamingGlobalConfigHandle globalConfigHandle) {
+      StreamingGlobalConfigHandle globalConfigHandle,
+      @Nullable ChannelCache channelCache) {
     this.clock = clock;
     this.clientId = clientId;
     this.isRunning = isRunning;
@@ -111,6 +114,7 @@ public final class StreamingWorkerStatusPages {
     this.workUnitExecutor = workUnitExecutor;
     this.statusPageDumper = statusPageDumper;
     globalConfigHandle.registerConfigObserver(globalConfig::set);
+    this.channelCache = channelCache;
   }
 
   public static StreamingWorkerStatusPages.Builder builder() {
@@ -168,6 +172,9 @@ private void addStreamingEngineStatusPages() {
           }
           writer.println(config.userWorkerJobSettings().toString());
         });
+    if (channelCache != null) {
+      statusPages.addStatusDataProvider("ChannelCache", "ChannelCache", channelCache);
+    }
   }
 
   private boolean isStreamingEngine() {
@@ -276,6 +283,8 @@ public interface Builder {
 
     Builder setGlobalConfigHandle(StreamingGlobalConfigHandle globalConfigHandle);
 
+    Builder setChannelCache(@Nullable ChannelCache channelCache);
+
     StreamingWorkerStatusPages build();
   }
 }
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
index 7d199afc0861..4ffd2861b483 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
@@ -20,6 +20,7 @@
 import com.google.auto.value.AutoValue;
 import java.util.Optional;
 import java.util.function.Function;
+import java.util.function.Supplier;
 import org.apache.beam.runners.dataflow.worker.windmill.CloudWindmillServiceV1Alpha1Grpc.CloudWindmillServiceV1Alpha1Stub;
 import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints.Endpoint;
 import org.apache.beam.sdk.annotations.Internal;
@@ -36,7 +37,7 @@ public static WindmillConnection from(
 
     windmillEndpoint.workerToken().ifPresent(windmillWorkerConnection::setBackendWorkerToken);
     windmillEndpoint.directEndpoint().ifPresent(windmillWorkerConnection::setDirectEndpoint);
-    windmillWorkerConnection.setStub(endpointToStubFn.apply(windmillEndpoint));
+    windmillWorkerConnection.setStubSupplier(() -> endpointToStubFn.apply(windmillEndpoint));
 
     return windmillWorkerConnection.build();
   }
@@ -48,17 +49,22 @@ public static Builder builder() {
 
   public abstract String backendWorkerToken();
 
-  public abstract Optional<WindmillServiceAddress> directEndpoint();
+  abstract Optional<WindmillServiceAddress> directEndpoint();
 
-  public abstract CloudWindmillServiceV1Alpha1Stub stub();
+  abstract Supplier<CloudWindmillServiceV1Alpha1Stub> stubSupplier();
+
+  public final CloudWindmillServiceV1Alpha1Stub currentStub() {
+    return stubSupplier().get();
+  }
 
   @AutoValue.Builder
   public abstract static class Builder {
     abstract Builder setBackendWorkerToken(String backendWorkerToken);
 
-    public abstract Builder setDirectEndpoint(WindmillServiceAddress value);
+    abstract Builder setDirectEndpoint(WindmillServiceAddress value);
 
-    public abstract Builder setStub(CloudWindmillServiceV1Alpha1Stub stub);
+    public abstract Builder setStubSupplier(
+        Supplier<CloudWindmillServiceV1Alpha1Stub> stubSupplier);
 
     public abstract WindmillConnection build();
   }
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClient.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClient.java
index 234888831779..3603bacf461a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClient.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClient.java
@@ -17,8 +17,8 @@
  */
 package org.apache.beam.runners.dataflow.worker.windmill.client.grpc;
 
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.LOCALHOST;
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.localhostChannel;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.LOCALHOST;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.localhostChannel;
 
 import com.google.auto.value.AutoValue;
 import java.util.List;
@@ -164,15 +164,6 @@ private synchronized <T> T randomlySelectNextStub(List<T> stubs) {
     return stubs.get(rand.nextInt(stubs.size()));
   }
 
-  /**
-   * Returns whether the {@link DispatcherStubs} have been set. Once initially set, {@link
-   * #dispatcherStubs} will always have a value as empty updates will trigger an {@link
-   * IllegalStateException}.
-   */
-  public boolean hasInitializedEndpoints() {
-    return dispatcherStubs.get().hasInitializedEndpoints();
-  }
-
   public void onJobConfig(StreamingGlobalConfig config) {
     if (config.windmillServiceEndpoints().isEmpty()) {
       LOG.warn("Dispatcher client received empty windmill service endpoints from global config");
@@ -282,14 +273,6 @@ private static CloudWindmillMetadataServiceV1Alpha1Stub createWindmillMetadataSe
           WindmillServiceAddress.create(endpoint));
     }
 
-    private int size() {
-      return dispatcherEndpoints().size();
-    }
-
-    private boolean hasInitializedEndpoints() {
-      return size() > 0;
-    }
-
     abstract ImmutableSet<HostAndPort> dispatcherEndpoints();
 
     abstract ImmutableList<CloudWindmillServiceV1Alpha1Stub> windmillServiceStubs();
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServer.java
index 8fe1f14caa77..e4a7e1a2f92c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServer.java
@@ -17,9 +17,9 @@
  */
 package org.apache.beam.runners.dataflow.worker.windmill.client.grpc;
 
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.LOCALHOST;
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.inProcessChannel;
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.localhostChannel;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.LOCALHOST;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.inProcessChannel;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.localhostChannel;
 
 import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 import java.io.IOException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java
index b0172cdbd6ff..d32cdcfac3cc 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillStreamFactory.java
@@ -230,7 +230,7 @@ public GetWorkStream createDirectGetWorkStream(
       WorkItemScheduler workItemScheduler) {
     return GrpcDirectGetWorkStream.create(
         connection.backendWorkerToken(),
-        responseObserver -> connection.stub().getWorkStream(responseObserver),
+        responseObserver -> connection.currentStub().getWorkStream(responseObserver),
         request,
         grpcBackOff.get(),
         newStreamObserverFactory(),
@@ -265,7 +265,7 @@ public GetDataStream createDirectGetDataStream(
       WindmillConnection connection, ThrottleTimer getDataThrottleTimer) {
     return GrpcGetDataStream.create(
         connection.backendWorkerToken(),
-        responseObserver -> connection.stub().getDataStream(responseObserver),
+        responseObserver -> connection.currentStub().getDataStream(responseObserver),
         grpcBackOff.get(),
         newStreamObserverFactory(),
         streamRegistry,
@@ -297,7 +297,7 @@ public CommitWorkStream createDirectCommitWorkStream(
       WindmillConnection connection, ThrottleTimer commitWorkThrottleTimer) {
     return GrpcCommitWorkStream.create(
         connection.backendWorkerToken(),
-        responseObserver -> connection.stub().commitWorkStream(responseObserver),
+        responseObserver -> connection.currentStub().commitWorkStream(responseObserver),
         grpcBackOff.get(),
         newStreamObserverFactory(),
         streamRegistry,
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
index 9a401690ca23..d34612725a1d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
@@ -17,13 +17,16 @@
  */
 package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
 
+import static org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.Kind.AUTHENTICATED_GCP_SERVICE_ADDRESS;
+
 import java.io.PrintWriter;
 import java.util.concurrent.Executor;
 import java.util.concurrent.Executors;
 import java.util.concurrent.TimeUnit;
-import java.util.function.Function;
+import javax.annotation.concurrent.GuardedBy;
 import javax.annotation.concurrent.ThreadSafe;
 import org.apache.beam.runners.dataflow.worker.status.StatusDataProvider;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
 import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
 import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
@@ -34,6 +37,7 @@
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.cache.RemovalListeners;
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.MoreExecutors;
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.checkerframework.checker.nullness.qual.MonotonicNonNull;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -48,10 +52,16 @@
 @ThreadSafe
 public final class ChannelCache implements StatusDataProvider {
   private static final Logger LOG = LoggerFactory.getLogger(ChannelCache.class);
+  private static final String DIRECTPATH = "Directpath";
+  private static final String CLOUDPATH = "Cloudpath";
   private final LoadingCache<WindmillServiceAddress, ManagedChannel> channelCache;
 
+  @GuardedBy("this")
+  @MonotonicNonNull
+  private UserWorkerGrpcFlowControlSettings currentFlowControlSettings = null;
+
   private ChannelCache(
-      Function<WindmillServiceAddress, ManagedChannel> channelFactory,
+      WindmillChannelFactory channelFactory,
       RemovalListener<WindmillServiceAddress, ManagedChannel> onChannelRemoved,
       Executor channelCloser) {
     this.channelCache =
@@ -61,13 +71,25 @@ private ChannelCache(
                 new CacheLoader<WindmillServiceAddress, ManagedChannel>() {
                   @Override
                   public ManagedChannel load(WindmillServiceAddress key) {
-                    return channelFactory.apply(key);
+                    return channelFactory.create(resolveFlowControlSettings(key.getKind()), key);
+                  }
+
+                  private UserWorkerGrpcFlowControlSettings resolveFlowControlSettings(
+                      WindmillServiceAddress.Kind addressType) {
+                    synchronized (ChannelCache.this) {
+                      if (currentFlowControlSettings == null) {
+                        return addressType == AUTHENTICATED_GCP_SERVICE_ADDRESS
+                            ? WindmillChannels.DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS
+                            : WindmillChannels.DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS;
+                      }
+
+                      return currentFlowControlSettings;
+                    }
                   }
                 });
   }
 
-  public static ChannelCache create(
-      Function<WindmillServiceAddress, ManagedChannel> channelFactory) {
+  public static ChannelCache create(WindmillChannelFactory channelFactory) {
     return new ChannelCache(
         channelFactory,
         // Shutdown the channels as they get removed from the cache, so they do not leak.
@@ -78,7 +100,7 @@ public static ChannelCache create(
 
   @VisibleForTesting
   public static ChannelCache forTesting(
-      Function<WindmillServiceAddress, ManagedChannel> channelFactory, Runnable onChannelShutdown) {
+      WindmillChannelFactory channelFactory, Runnable onChannelShutdown) {
     return new ChannelCache(
         channelFactory,
         // Shutdown the channels as they get removed from the cache, so they do not leak.
@@ -98,7 +120,7 @@ private static void shutdownChannel(ManagedChannel channel) {
     try {
       channel.awaitTermination(10, TimeUnit.SECONDS);
     } catch (InterruptedException e) {
-      LOG.error("Couldn't close gRPC channel={}", channel, e);
+      LOG.error("Couldn't gracefully close gRPC channel={}", channel, e);
     }
     channel.shutdownNow();
   }
@@ -107,6 +129,18 @@ public ManagedChannel get(WindmillServiceAddress windmillServiceAddress) {
     return channelCache.getUnchecked(windmillServiceAddress);
   }
 
+  public synchronized void consumeFlowControlSettings(
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    if (!flowControlSettings.equals(currentFlowControlSettings)) {
+      // Refreshing the cache will asynchronously terminate the old channels via the removalListener
+      // and return a newly created one on the next Cache.load(address). This could be expensive so
+      // only do it when we have received new flow control settings.
+      LOG.debug("Updating flow control settings {}.", flowControlSettings);
+      currentFlowControlSettings = flowControlSettings;
+      channelCache.asMap().keySet().forEach(channelCache::refresh);
+    }
+  }
+
   public void remove(WindmillServiceAddress windmillServiceAddress) {
     channelCache.invalidate(windmillServiceAddress);
   }
@@ -128,12 +162,28 @@ boolean isEmpty() {
 
   @Override
   public void appendSummaryHtml(PrintWriter writer) {
+    synchronized (this) {
+      if (currentFlowControlSettings == null) {
+        writer.format(
+            "Directpath gRPC flow control settings: [%s]",
+            WindmillChannels.DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS);
+        writer.format(
+            "Cloudpath gRPC flow control settings: [%s]",
+            WindmillChannels.DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS);
+      } else {
+        writer.format("gRPC flow control settings: [%s]", currentFlowControlSettings);
+      }
+    }
     writer.write("Active gRPC Channels:<br>");
     channelCache
         .asMap()
         .forEach(
             (address, channel) -> {
-              writer.format("Address: [%s]; Channel: [%s].", address, channel);
+              writer.format(
+                  "Address: [%s]; Channel: [%s]; AddressType:[%s].",
+                  address,
+                  channel,
+                  address.getKind() == AUTHENTICATED_GCP_SERVICE_ADDRESS ? DIRECTPATH : CLOUDPATH);
               writer.write("<br>");
             });
   }
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/IsolationChannel.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/IsolationChannel.java
index d78dd421dcc8..e0cb032a5f87 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/IsolationChannel.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/IsolationChannel.java
@@ -34,7 +34,6 @@
 import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Metadata;
 import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.MethodDescriptor;
 import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Status;
-import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
 import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -48,6 +47,7 @@ public class IsolationChannel extends ManagedChannel {
   private static final Logger LOG = LoggerFactory.getLogger(IsolationChannel.class);
 
   private final Supplier<ManagedChannel> channelFactory;
+  private final int onReadyThresholdBytes;
 
   @GuardedBy("this")
   private final List<ManagedChannel> channelCache;
@@ -55,24 +55,29 @@ public class IsolationChannel extends ManagedChannel {
   @GuardedBy("this")
   private final Set<ManagedChannel> usedChannels;
 
-  @GuardedBy("this")
-  private boolean shutdownStarted = false;
-
   private final String authority;
 
-  // Expected that supplier returns channels to the same endpoint with the same authority.
-  public static IsolationChannel create(Supplier<ManagedChannel> channelFactory) {
-    return new IsolationChannel(channelFactory);
-  }
+  @GuardedBy("this")
+  private boolean shutdownStarted = false;
 
-  @VisibleForTesting
-  IsolationChannel(Supplier<ManagedChannel> channelFactory) {
+  private IsolationChannel(Supplier<ManagedChannel> channelFactory, int onReadyThresholdBytes) {
     this.channelFactory = channelFactory;
     this.channelCache = new ArrayList<>();
     ManagedChannel channel = channelFactory.get();
     this.authority = channel.authority();
     this.channelCache.add(channel);
     this.usedChannels = new HashSet<>();
+    this.onReadyThresholdBytes = onReadyThresholdBytes;
+  }
+
+  // Expected that supplier returns channels to the same endpoint with the same authority.
+  public static IsolationChannel create(Supplier<ManagedChannel> channelFactory) {
+    return new IsolationChannel(channelFactory, -1);
+  }
+
+  public static IsolationChannel create(
+      Supplier<ManagedChannel> channelFactory, int onReadyThresholdBytes) {
+    return new IsolationChannel(channelFactory, onReadyThresholdBytes);
   }
 
   @Override
@@ -85,7 +90,14 @@ public String authority() {
   public <ReqT, RespT> ClientCall<ReqT, RespT> newCall(
       MethodDescriptor<ReqT, RespT> methodDescriptor, CallOptions callOptions) {
     ManagedChannel channel = getChannel();
-    return new ReleasingClientCall<>(this, channel, channel.newCall(methodDescriptor, callOptions));
+    return new ReleasingClientCall<>(
+        this,
+        channel,
+        channel.newCall(
+            methodDescriptor,
+            onReadyThresholdBytes > 0
+                ? callOptions.withOnReadyThreshold(onReadyThresholdBytes)
+                : callOptions));
   }
 
   private synchronized ManagedChannel getChannel() {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
index 8ca962e95604..3a64528ca98d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
@@ -17,120 +17,13 @@
  */
 package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
 
-import java.util.concurrent.TimeUnit;
-import javax.net.ssl.SSLException;
-import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
 import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
-import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
 import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
-import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
 
-/** Utility class used to create different RPC Channels. */
-public final class WindmillChannelFactory {
-  public static final String LOCALHOST = "localhost";
-  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
-  // 10MiB.
-  private static final int WINDMILL_MAX_FLOW_CONTROL_WINDOW =
-      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
-
-  private WindmillChannelFactory() {}
-
-  public static ManagedChannel inProcessChannel(String channelName) {
-    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
-  }
-
-  public static Channel localhostChannel(int port) {
-    return NettyChannelBuilder.forAddress(LOCALHOST, port)
-        .maxInboundMessageSize(Integer.MAX_VALUE)
-        .negotiationType(NegotiationType.PLAINTEXT)
-        .build();
-  }
-
-  public static ManagedChannel remoteChannel(
-      WindmillServiceAddress windmillServiceAddress, int windmillServiceRpcChannelTimeoutSec) {
-    switch (windmillServiceAddress.getKind()) {
-      case GCP_SERVICE_ADDRESS:
-        return remoteChannel(
-            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
-      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
-        return remoteDirectChannel(
-            windmillServiceAddress.authenticatedGcpServiceAddress(),
-            windmillServiceRpcChannelTimeoutSec);
-      default:
-        throw new UnsupportedOperationException(
-            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
-                + " WindmillServiceAddresses.");
-    }
-  }
-
-  private static ManagedChannel remoteDirectChannel(
-      AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
-      int windmillServiceRpcChannelTimeoutSec) {
-    return withDefaultChannelOptions(
-            NettyChannelBuilder.forAddress(
-                    authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
-                    // Ports are required for direct channels.
-                    authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
-                    new AltsChannelCredentials.Builder().build())
-                .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
-            windmillServiceRpcChannelTimeoutSec)
-        .build();
-  }
-
-  public static ManagedChannel remoteChannel(
-      HostAndPort endpoint, int windmillServiceRpcChannelTimeoutSec) {
-    return withDefaultChannelOptions(
-            NettyChannelBuilder.forAddress(
-                endpoint.getHost(),
-                endpoint.hasPort()
-                    ? endpoint.getPort()
-                    : WindmillEndpoints.DEFAULT_WINDMILL_SERVICE_PORT),
-            windmillServiceRpcChannelTimeoutSec)
-        .negotiationType(NegotiationType.TLS)
-        .sslContext(dataflowGrpcSslContext(endpoint))
-        .build();
-  }
-
-  @SuppressWarnings("nullness")
-  private static SslContext dataflowGrpcSslContext(HostAndPort endpoint) {
-    try {
-      // Set ciphers(null) to not use GCM, which is disabled for Dataflow
-      // due to it being horribly slow.
-      return GrpcSslContexts.forClient().ciphers(null).build();
-    } catch (SSLException sslException) {
-      throw new WindmillChannelCreationException(endpoint, sslException);
-    }
-  }
-
-  private static NettyChannelBuilder withDefaultChannelOptions(
-      NettyChannelBuilder channelBuilder, int windmillServiceRpcChannelTimeoutSec) {
-    if (windmillServiceRpcChannelTimeoutSec > 0) {
-      channelBuilder
-          .keepAliveTime(windmillServiceRpcChannelTimeoutSec, TimeUnit.SECONDS)
-          .keepAliveTimeout(windmillServiceRpcChannelTimeoutSec, TimeUnit.SECONDS)
-          .keepAliveWithoutCalls(true);
-    }
-
-    return channelBuilder
-        .maxInboundMessageSize(Integer.MAX_VALUE)
-        .maxTraceEvents(MAX_REMOTE_TRACE_EVENTS)
-        // 1MiB
-        .maxInboundMetadataSize(1024 * 1024)
-        .flowControlWindow(WINDMILL_MAX_FLOW_CONTROL_WINDOW);
-  }
-
-  private static class WindmillChannelCreationException extends IllegalStateException {
-    private WindmillChannelCreationException(HostAndPort endpoint, SSLException sourceException) {
-      super(
-          String.format("Exception thrown when trying to create channel to %s", endpoint),
-          sourceException);
-    }
-  }
+@FunctionalInterface
+public interface WindmillChannelFactory {
+  ManagedChannel create(
+      UserWorkerGrpcFlowControlSettings flowControlSettings,
+      WindmillServiceAddress windmillServiceAddress);
 }
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
new file mode 100644
index 000000000000..375cd354ac90
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+  public static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+  // 100MiB.
+  private static final int DIRECTPATH_MIN_FLOW_CONTROL_WINDOW =
+      WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10;
+  // 100MiB.
+  private static final int DIRECTPATH_ON_READY_THRESHOLD = WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10;
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  public static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(DIRECTPATH_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(DIRECTPATH_ON_READY_THRESHOLD)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right away. There are internal pushback mechanisms in
+          // the user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  private static ManagedChannel remoteDirectChannel(
+      AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
+                    // Ports are required for direct channels.
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
+                    new AltsChannelCredentials.Builder().build())
+                .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
+            windmillServiceRpcChannelTimeoutSec);
+
+    return withFlowControlSettings(
+            channelBuilder, flowControlSettings, DIRECTPATH_MIN_FLOW_CONTROL_WINDOW)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      HostAndPort endpoint,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
+                NettyChannelBuilder.forAddress(
+                    endpoint.getHost(),
+                    endpoint.hasPort()
+                        ? endpoint.getPort()
+                        : WindmillEndpoints.DEFAULT_WINDMILL_SERVICE_PORT),
+                windmillServiceRpcChannelTimeoutSec)
+            .negotiationType(NegotiationType.TLS)
+            .sslContext(dataflowGrpcSslContext(endpoint));
+
+    return withFlowControlSettings(
+            channelBuilder, flowControlSettings, WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+        .build();
+  }
+
+  private static NettyChannelBuilder withFlowControlSettings(
+      NettyChannelBuilder channelBuilder,
+      UserWorkerGrpcFlowControlSettings flowControlSettings,
+      int defaultFlowControlBytes) {
+    int flowControlWindowSizeBytes =
+        Math.max(defaultFlowControlBytes, flowControlSettings.getFlowControlWindowBytes());
+    return flowControlSettings.getEnableAutoFlowControl()
+        // Enable auto flow control with an initial window of flowControlWindowSizeBytes. gRPC may
+        // resize this value based on throughput.
+        ? channelBuilder.initialFlowControlWindow(flowControlWindowSizeBytes)
+        : channelBuilder.flowControlWindow(flowControlWindowSizeBytes);
+  }
+
+  @SuppressWarnings("nullness")
+  private static SslContext dataflowGrpcSslContext(HostAndPort endpoint) {
+    try {
+      // Set ciphers(null) to not use GCM, which is disabled for Dataflow
+      // due to it being horribly slow.
+      return GrpcSslContexts.forClient().ciphers(null).build();
+    } catch (SSLException sslException) {
+      throw new WindmillChannelCreationException(endpoint, sslException);
+    }
+  }
+
+  private static NettyChannelBuilder withDefaultChannelOptions(
+      NettyChannelBuilder channelBuilder, int windmillServiceRpcChannelTimeoutSec) {
+    if (windmillServiceRpcChannelTimeoutSec > 0) {
+      channelBuilder
+          .keepAliveTime(windmillServiceRpcChannelTimeoutSec, TimeUnit.SECONDS)
+          .keepAliveTimeout(windmillServiceRpcChannelTimeoutSec, TimeUnit.SECONDS)
+          .keepAliveWithoutCalls(true);
+    }
+
+    return channelBuilder
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .maxTraceEvents(MAX_REMOTE_TRACE_EVENTS)
+        .maxInboundMetadataSize(MAX_INBOUND_METADATA_SIZE_BYTES);
+  }
+
+  private static class WindmillChannelCreationException extends IllegalStateException {
+    private WindmillChannelCreationException(HostAndPort endpoint, SSLException sourceException) {
+      super(
+          String.format("Exception thrown when trying to create channel to %s", endpoint),
+          sourceException);
+    }
+  }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
index 98ff93733d66..6de6b6337539 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
@@ -17,16 +17,12 @@
  */
 package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
 
-import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.remoteChannel;
+import static org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels.remoteChannel;
 
 import com.google.auth.Credentials;
-import java.util.function.Function;
 import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
-import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
-import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
 
 public class WindmillStubFactoryFactoryImpl implements WindmillStubFactoryFactory {
-
   private final int windmillServiceRpcChannelAliveTimeoutSec;
   private final Credentials gcpCredential;
 
@@ -38,17 +34,23 @@ public WindmillStubFactoryFactoryImpl(DataflowWorkerHarnessOptions workerOptions
 
   @Override
   public WindmillStubFactory makeWindmillStubFactory(boolean useIsolatedChannels) {
-    Function<WindmillServiceAddress, ManagedChannel> channelFactory =
-        serviceAddress -> remoteChannel(serviceAddress, windmillServiceRpcChannelAliveTimeoutSec);
     ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
+            (flowControlSettings, serviceAddress) ->
                 // IsolationChannel will create and manage separate RPC channels to the same
                 // serviceAddress via calling the channelFactory, else just directly return the
                 // RPC channel.
                 useIsolatedChannels
-                    ? IsolationChannel.create(() -> channelFactory.apply(serviceAddress))
-                    : channelFactory.apply(serviceAddress));
+                    ? IsolationChannel.create(
+                        () ->
+                            remoteChannel(
+                                serviceAddress.getServiceAddress(),
+                                windmillServiceRpcChannelAliveTimeoutSec,
+                                flowControlSettings))
+                    : remoteChannel(
+                        serviceAddress.getServiceAddress(),
+                        windmillServiceRpcChannelAliveTimeoutSec,
+                        flowControlSettings));
     return ChannelCachingRemoteStubFactory.create(gcpCredential, channelCache);
   }
 }
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java
index 539520a2edc5..c60228f336e9 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java
@@ -132,7 +132,7 @@
 import org.apache.beam.runners.dataflow.worker.windmill.Windmill.WatermarkHold;
 import org.apache.beam.runners.dataflow.worker.windmill.Windmill.WorkItemCommitRequest;
 import org.apache.beam.runners.dataflow.worker.windmill.client.getdata.FakeGetDataClient;
-import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory;
+import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels;
 import org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactoryFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.work.refresh.HeartbeatSender;
@@ -883,8 +883,7 @@ private StreamingDataflowWorker makeWorker(
             new FakeWindmillStubFactoryFactory(
                 new FakeWindmillStubFactory(
                     () ->
-                        WindmillChannelFactory.inProcessChannel(
-                            "StreamingDataflowWorkerTestChannel"))));
+                        WindmillChannels.inProcessChannel("StreamingDataflowWorkerTestChannel"))));
     this.computationStateCache = worker.getComputationStateCache();
     return worker;
   }
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/FanOutStreamingEngineWorkerHarnessTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/FanOutStreamingEngineWorkerHarnessTest.java
index ff98b7ed3221..b3aa321b9808 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/FanOutStreamingEngineWorkerHarnessTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/FanOutStreamingEngineWorkerHarnessTest.java
@@ -51,7 +51,7 @@
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.GrpcDispatcherClient;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.GrpcWindmillStreamFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCachingStubFactory;
-import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory;
+import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels;
 import org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactoryFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.work.WorkItemScheduler;
@@ -81,7 +81,7 @@
 public class FanOutStreamingEngineWorkerHarnessTest {
   private static final String CHANNEL_NAME = "FanOutStreamingEngineWorkerHarnessTest";
   private static final WindmillServiceAddress DEFAULT_WINDMILL_SERVICE_ADDRESS =
-      WindmillServiceAddress.create(HostAndPort.fromParts(WindmillChannelFactory.LOCALHOST, 443));
+      WindmillServiceAddress.create(HostAndPort.fromParts(WindmillChannels.LOCALHOST, 443));
   private static final ImmutableMap<String, WorkerMetadataResponse.Endpoint> DEFAULT =
       ImmutableMap.of(
           "global_data",
@@ -107,7 +107,7 @@ public class FanOutStreamingEngineWorkerHarnessTest {
       spy(GrpcWindmillStreamFactory.of(JOB_HEADER).build());
   private final ChannelCachingStubFactory stubFactory =
       new FakeWindmillStubFactory(
-          () -> grpcCleanup.register(WindmillChannelFactory.inProcessChannel(CHANNEL_NAME)));
+          () -> grpcCleanup.register(WindmillChannels.inProcessChannel(CHANNEL_NAME)));
   private final GrpcDispatcherClient dispatcherClient =
       GrpcDispatcherClient.forTesting(
           PipelineOptionsFactory.as(DataflowWorkerHarnessOptions.class),
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSenderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSenderTest.java
index 21d47b2cd835..e570a83e3457 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSenderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/harness/WindmillStreamSenderTest.java
@@ -82,7 +82,7 @@ public void setUp() {
     grpcCleanup.register(inProcessChannel);
     connection =
         WindmillConnection.builder()
-            .setStub(CloudWindmillServiceV1Alpha1Grpc.newStub(inProcessChannel))
+            .setStubSupplier(() -> CloudWindmillServiceV1Alpha1Grpc.newStub(inProcessChannel))
             .build();
   }
 
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDirectGetWorkStreamTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDirectGetWorkStreamTest.java
index 6cb68191f2da..b9592e9aacc4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDirectGetWorkStreamTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDirectGetWorkStreamTest.java
@@ -143,7 +143,8 @@ private GrpcDirectGetWorkStream createGetWorkStream(
                 .build()
                 .createDirectGetWorkStream(
                     WindmillConnection.builder()
-                        .setStub(CloudWindmillServiceV1Alpha1Grpc.newStub(inProcessChannel))
+                        .setStubSupplier(
+                            () -> CloudWindmillServiceV1Alpha1Grpc.newStub(inProcessChannel))
                         .build(),
                     Windmill.GetWorkRequest.newBuilder()
                         .setClientId(TEST_JOB_HEADER.getClientId())
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServerTest.java
index bdb482ba768a..211e926a073c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServerTest.java
@@ -72,7 +72,7 @@
 import org.apache.beam.runners.dataflow.worker.windmill.client.WindmillStream.GetDataStream;
 import org.apache.beam.runners.dataflow.worker.windmill.client.WindmillStream.GetWorkStream;
 import org.apache.beam.runners.dataflow.worker.windmill.client.WindmillStreamShutdownException;
-import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory;
+import org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannels;
 import org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactory;
 import org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactoryFactory;
 import org.apache.beam.vendor.grpc.v1p69p0.com.google.protobuf.ByteString;
@@ -156,7 +156,7 @@ private void startServerAndClient(List<String> experiments) throws Exception {
                 new FakeWindmillStubFactory(
                     () -> {
                       ManagedChannel channel =
-                          grpcCleanup.register(WindmillChannelFactory.inProcessChannel(name));
+                          grpcCleanup.register(WindmillChannels.inProcessChannel(name));
                       openedChannels.add(channel);
                       return channel;
                     })));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
index 704a9c9fdb2e..311bed75ccc7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
@@ -18,6 +18,7 @@
 package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
 
 import static com.google.common.truth.Truth.assertThat;
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertSame;
 import static org.junit.Assert.assertTrue;
 import static org.mockito.ArgumentMatchers.eq;
@@ -25,11 +26,18 @@
 import static org.mockito.Mockito.spy;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
 
+import java.util.Queue;
 import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Function;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
 import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
 import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterables;
 import org.junit.After;
 import org.junit.Rule;
 import org.junit.Test;
@@ -45,7 +53,7 @@ public class ChannelCacheTest {
 
   private static ChannelCache newCache(
       Function<WindmillServiceAddress, ManagedChannel> channelFactory) {
-    return ChannelCache.forTesting(channelFactory, () -> {});
+    return ChannelCache.forTesting((ignored, address) -> channelFactory.apply(address), () -> {});
   }
 
   @After
@@ -56,7 +64,7 @@ public void cleanUp() {
   }
 
   private ManagedChannel newChannel(String channelName) {
-    return WindmillChannelFactory.inProcessChannel(channelName);
+    return WindmillChannels.inProcessChannel(channelName);
   }
 
   @Test
@@ -108,7 +116,8 @@ public void testRemoveAndClose() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (ignoredFlowControlSettings, ignoredServiceAddress) -> newChannel(channelName),
+            notifyWhenChannelClosed::countDown);
 
     WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
     ManagedChannel cachedChannel = cache.get(someAddress);
@@ -132,7 +141,8 @@ public void testClear() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (ignoredFlowControlSettings, ignoredServiceAddress) -> newChannel(channelName),
+            notifyWhenChannelClosed::countDown);
     WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
     ManagedChannel cachedChannel = cache.get(someAddress);
     cache.clear();
@@ -140,4 +150,89 @@ public void testClear() throws InterruptedException {
     assertTrue(cache.isEmpty());
     assertTrue(cachedChannel.isShutdown());
   }
+
+  @Test
+  public void testConsumeFlowControlSettings() throws InterruptedException {
+    String channelName = "channel";
+    CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    AtomicReference<UserWorkerGrpcFlowControlSettings> consumedFlowControlSettings =
+        new AtomicReference<>();
+    cache =
+        ChannelCache.forTesting(
+            (newFlowControlSettings, ignoredServiceAddress) -> {
+              ManagedChannel channel = newChannel(channelName);
+              newChannelsCreated.incrementAndGet();
+              consumedFlowControlSettings.set(newFlowControlSettings);
+              return channel;
+            },
+            notifyWhenChannelClosed::countDown);
+    WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
+    when(someAddress.getKind())
+        .thenReturn(WindmillServiceAddress.Kind.AUTHENTICATED_GCP_SERVICE_ADDRESS);
+
+    // Load the cache w/ this first get.
+    ManagedChannel cachedChannel = cache.get(someAddress);
+
+    // Load flow control settings.
+    UserWorkerGrpcFlowControlSettings flowControlSettings =
+        UserWorkerGrpcFlowControlSettings.newBuilder()
+            .setEnableAutoFlowControl(true)
+            .setOnReadyThresholdBytes(1)
+            .setFlowControlWindowBytes(1)
+            .build();
+    cache.consumeFlowControlSettings(flowControlSettings);
+
+    // This get should reload the cache, since flow control settings have changed.
+    ManagedChannel reloadedChannel = cache.get(someAddress);
+    notifyWhenChannelClosed.await();
+    assertThat(cachedChannel).isNotSameInstanceAs(reloadedChannel);
+    assertTrue(cachedChannel.isShutdown());
+    assertFalse(reloadedChannel.isShutdown());
+    assertThat(newChannelsCreated.get()).isEqualTo(2);
+    assertThat(cache.get(someAddress)).isSameInstanceAs(reloadedChannel);
+    assertThat(consumedFlowControlSettings.get()).isEqualTo(flowControlSettings);
+  }
+
+  @Test
+  public void testConsumeFlowControlSettings_sameFlowControlSettings() {
+    String channelName = "channel";
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    UserWorkerGrpcFlowControlSettings flowControlSettings =
+        UserWorkerGrpcFlowControlSettings.newBuilder()
+            .setEnableAutoFlowControl(true)
+            .setOnReadyThresholdBytes(1)
+            .setFlowControlWindowBytes(1)
+            .build();
+    Queue<UserWorkerGrpcFlowControlSettings> consumedFlowControlSettings =
+        new LinkedBlockingQueue<>();
+    cache =
+        ChannelCache.forTesting(
+            (newFlowControlSettings, ignoredServiceAddress) -> {
+              ManagedChannel channel = newChannel(channelName);
+              newChannelsCreated.incrementAndGet();
+              consumedFlowControlSettings.add(newFlowControlSettings);
+              return channel;
+            },
+            () -> {});
+    // Load flow control settings.
+    cache.consumeFlowControlSettings(flowControlSettings);
+    WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
+    when(someAddress.getKind())
+        .thenReturn(WindmillServiceAddress.Kind.AUTHENTICATED_GCP_SERVICE_ADDRESS);
+    // Load the cache w/ this first get.
+    ManagedChannel cachedChannel = cache.get(someAddress);
+
+    // Load the same flow control settings.
+    cache.consumeFlowControlSettings(flowControlSettings);
+    // Because the flow control settings are the same, cache should not reload the value for the
+    // same address.
+    ManagedChannel reloadedChannel = cache.get(someAddress);
+    assertThat(cachedChannel).isSameInstanceAs(reloadedChannel);
+    assertFalse(cachedChannel.isShutdown());
+    assertThat(newChannelsCreated.get()).isEqualTo(1);
+    assertThat(consumedFlowControlSettings.size()).isEqualTo(1);
+    assertThat(Iterables.getOnlyElement(consumedFlowControlSettings))
+        .isEqualTo(flowControlSettings);
+  }
 }
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactory.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactory.java
index b0897574d9ac..79297e51a55f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactory.java
@@ -31,7 +31,9 @@ public final class FakeWindmillStubFactory implements ChannelCachingStubFactory
   private final ChannelCache channelCache;
 
   public FakeWindmillStubFactory(Supplier<ManagedChannel> channelFactory) {
-    this.channelCache = ChannelCache.forTesting(ignored -> channelFactory.get(), () -> {});
+    this.channelCache =
+        ChannelCache.forTesting(
+            (ignoredFlowControlSettings, ignoredServiceAddress) -> channelFactory.get(), () -> {});
   }
 
   @Override


Reviews:
user: login: scwhittle
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T09:31:55Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T17:48:27Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T17:52:28Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:12:09Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:12:54Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:13:18Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:13:24Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:25:07Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:28:15Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:29:28Z
user: login: scwhittle
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T15:49:42Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T19:05:30Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T22:55:13Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T22:55:27Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T22:56:41Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T22:56:49Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:44:55Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:46:18Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:55:32Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:56:06Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T07:32:43Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T07:35:47Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T07:35:54Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T07:35:56Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T07:40:28Z
user: login: scwhittle
body: LGTM, just minor cleanup comment
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T13:44:36Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T21:30:06Z
user: login: damccorm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:04:00Z
user: login: m-trieu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T21:35:21Z
user: login: damccorm
body: Thanks
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T14:00:03Z

Review comments:
diff_hunk: @@ -621,18 +622,40 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            serviceAddress -> {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: scwhittle
body: could we pass in the cache's view of the settings instead?

It seems like it could be simpler to just have the single watcher for settings instead of also fetching here.
created_at: 2025-04-04T09:16:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,17 +49,21 @@ public static Builder builder() {
 
   public abstract String backendWorkerToken();
 
-  public abstract Optional<WindmillServiceAddress> directEndpoint();
+  abstract Optional<WindmillServiceAddress> directEndpoint();
 
-  public abstract CloudWindmillServiceV1Alpha1Stub stub();
+  abstract Supplier<CloudWindmillServiceV1Alpha1Stub> stubFactory();
+
+  public final CloudWindmillServiceV1Alpha1Stub newStub() {
+    return stubFactory().get();
+  }
 
   @AutoValue.Builder
   public abstract static class Builder {
     abstract Builder setBackendWorkerToken(String backendWorkerToken);
 
-    public abstract Builder setDirectEndpoint(WindmillServiceAddress value);
+    abstract Builder setDirectEndpoint(WindmillServiceAddress value);
 
-    public abstract Builder setStub(CloudWindmillServiceV1Alpha1Stub stub);
+    public abstract Builder setStubFactory(Supplier<CloudWindmillServiceV1Alpha1Stub> stubFactory);
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
user: login: scwhittle
body: should we name Supplier? factory implies it is making something new, where supplier could just be returning a cached stub (which is what we want if it hasn't been modified)
created_at: 2025-04-04T09:18:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,17 +49,21 @@ public static Builder builder() {
 
   public abstract String backendWorkerToken();
 
-  public abstract Optional<WindmillServiceAddress> directEndpoint();
+  abstract Optional<WindmillServiceAddress> directEndpoint();
 
-  public abstract CloudWindmillServiceV1Alpha1Stub stub();
+  abstract Supplier<CloudWindmillServiceV1Alpha1Stub> stubFactory();
+
+  public final CloudWindmillServiceV1Alpha1Stub newStub() {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
user: login: scwhittle
body: if it's a supplier, I think currentStub() or something could be better name since new implies it is freshly allocated and not reused
created_at: 2025-04-04T09:19:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -621,18 +622,40 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            serviceAddress -> {
+              // Always fetch the current flow control settings when we go to create the channel.
+              UserWorkerGrpcFlowControlSettings currentFlowControlSettings =
+                  configFetcher
+                      .getGlobalConfigHandle()
+                      .getConfig()
+                      .userWorkerJobSettings()
+                      .getFlowControlSettings();
+              // IsolationChannel will create and manage separate RPC channels to the same
+              // serviceAddress.
+              return IsolationChannel.create(
+                  () ->
+                      remoteChannel(
+                          serviceAddress,
+                          workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec(),
+                          currentFlowControlSettings),
+                  currentFlowControlSettings.getOnReadyThresholdBytes());
+            },
+            configFetcher
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: scwhittle
body: ditto seems simpler if we remove this constructor param, and have the registeredconfig observer trigger and update the flow control settings.
created_at: 2025-04-04T09:21:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -107,6 +122,23 @@ public ManagedChannel get(WindmillServiceAddress windmillServiceAddress) {
     return channelCache.getUnchecked(windmillServiceAddress);
   }
 
+  public synchronized void consumeFlowControlSettings(
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    if (!flowControlSettings.equals(currentFlowControlSettings)) {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: scwhittle
body: is there a way to check equilvalence instead of equals()? if field is explicilty 10MB versus default of 10MB we don't want to do anything either.
created_at: 2025-04-04T09:23:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -122,8 +135,7 @@ private static NettyChannelBuilder withDefaultChannelOptions(
         .maxInboundMessageSize(Integer.MAX_VALUE)
         .maxTraceEvents(MAX_REMOTE_TRACE_EVENTS)
         // 1MiB
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
user: login: scwhittle
body: rm comment
created_at: 2025-04-04T09:24:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,16 +78,22 @@ public static ManagedChannel remoteChannel(
 
   private static ManagedChannel remoteDirectChannel(
       AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
-      int windmillServiceRpcChannelTimeoutSec) {
-    return withDefaultChannelOptions(
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
             NettyChannelBuilder.forAddress(
                     authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
                     // Ports are required for direct channels.
                     authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
                     new AltsChannelCredentials.Builder().build())
                 .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
-            windmillServiceRpcChannelTimeoutSec)
-        .build();
+            windmillServiceRpcChannelTimeoutSec);
+    int flowControlWindowSizeBytes =
+        Math.max(WINDMILL_MAX_FLOW_CONTROL_WINDOW, flowControlSettings.getFlowControlWindowBytes());
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
user: login: scwhittle
body: seems like it should be WINDMILL_MIN_FLOW_CONTROL_WINDOW
created_at: 2025-04-04T09:27:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,6 +37,8 @@
 public final class WindmillChannelFactory {
   public static final String LOCALHOST = "localhost";
   private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
   // 10MiB.
   private static final int WINDMILL_MAX_FLOW_CONTROL_WINDOW =
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
user: login: scwhittle
body: Could add comment that this is chosen to be greater than 2 * max message size
created_at: 2025-04-04T09:28:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -621,18 +622,40 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            serviceAddress -> {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: m-trieu
body: done
created_at: 2025-04-08T17:48:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,17 +49,21 @@ public static Builder builder() {
 
   public abstract String backendWorkerToken();
 
-  public abstract Optional<WindmillServiceAddress> directEndpoint();
+  abstract Optional<WindmillServiceAddress> directEndpoint();
 
-  public abstract CloudWindmillServiceV1Alpha1Stub stub();
+  abstract Supplier<CloudWindmillServiceV1Alpha1Stub> stubFactory();
+
+  public final CloudWindmillServiceV1Alpha1Stub newStub() {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
user: login: m-trieu
body: done
created_at: 2025-04-08T17:52:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -107,6 +122,23 @@ public ManagedChannel get(WindmillServiceAddress windmillServiceAddress) {
     return channelCache.getUnchecked(windmillServiceAddress);
   }
 
+  public synchronized void consumeFlowControlSettings(
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    if (!flowControlSettings.equals(currentFlowControlSettings)) {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: m-trieu
body: done, i also added a default flow control setting for directpath.  equals in java compares the fields.
created_at: 2025-04-08T18:12:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,6 +37,8 @@
 public final class WindmillChannelFactory {
   public static final String LOCALHOST = "localhost";
   private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
   // 10MiB.
   private static final int WINDMILL_MAX_FLOW_CONTROL_WINDOW =
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
user: login: m-trieu
body: done
created_at: 2025-04-08T18:12:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,16 +78,22 @@ public static ManagedChannel remoteChannel(
 
   private static ManagedChannel remoteDirectChannel(
       AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
-      int windmillServiceRpcChannelTimeoutSec) {
-    return withDefaultChannelOptions(
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
             NettyChannelBuilder.forAddress(
                     authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
                     // Ports are required for direct channels.
                     authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
                     new AltsChannelCredentials.Builder().build())
                 .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
-            windmillServiceRpcChannelTimeoutSec)
-        .build();
+            windmillServiceRpcChannelTimeoutSec);
+    int flowControlWindowSizeBytes =
+        Math.max(WINDMILL_MAX_FLOW_CONTROL_WINDOW, flowControlSettings.getFlowControlWindowBytes());
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
user: login: m-trieu
body: done
created_at: 2025-04-08T18:13:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -122,8 +135,7 @@ private static NettyChannelBuilder withDefaultChannelOptions(
         .maxInboundMessageSize(Integer.MAX_VALUE)
         .maxTraceEvents(MAX_REMOTE_TRACE_EVENTS)
         // 1MiB
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannelFactory.java
user: login: m-trieu
body: done
created_at: 2025-04-08T18:13:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -621,18 +622,40 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            serviceAddress -> {
+              // Always fetch the current flow control settings when we go to create the channel.
+              UserWorkerGrpcFlowControlSettings currentFlowControlSettings =
+                  configFetcher
+                      .getGlobalConfigHandle()
+                      .getConfig()
+                      .userWorkerJobSettings()
+                      .getFlowControlSettings();
+              // IsolationChannel will create and manage separate RPC channels to the same
+              // serviceAddress.
+              return IsolationChannel.create(
+                  () ->
+                      remoteChannel(
+                          serviceAddress,
+                          workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec(),
+                          currentFlowControlSettings),
+                  currentFlowControlSettings.getOnReadyThresholdBytes());
+            },
+            configFetcher
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: m-trieu
body: done, also set some defaults
created_at: 2025-04-08T18:25:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -621,18 +622,40 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            serviceAddress -> {
+              // Always fetch the current flow control settings when we go to create the channel.
+              UserWorkerGrpcFlowControlSettings currentFlowControlSettings =
+                  configFetcher
+                      .getGlobalConfigHandle()
+                      .getConfig()
+                      .userWorkerJobSettings()
+                      .getFlowControlSettings();
+              // IsolationChannel will create and manage separate RPC channels to the same
+              // serviceAddress.
+              return IsolationChannel.create(
+                  () ->
+                      remoteChannel(
+                          serviceAddress,
+                          workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec(),
+                          currentFlowControlSettings),
+                  currentFlowControlSettings.getOnReadyThresholdBytes());
+            },
+            configFetcher
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: m-trieu
body: done
created_at: 2025-04-08T18:28:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,17 +49,21 @@ public static Builder builder() {
 
   public abstract String backendWorkerToken();
 
-  public abstract Optional<WindmillServiceAddress> directEndpoint();
+  abstract Optional<WindmillServiceAddress> directEndpoint();
 
-  public abstract CloudWindmillServiceV1Alpha1Stub stub();
+  abstract Supplier<CloudWindmillServiceV1Alpha1Stub> stubFactory();
+
+  public final CloudWindmillServiceV1Alpha1Stub newStub() {
+    return stubFactory().get();
+  }
 
   @AutoValue.Builder
   public abstract static class Builder {
     abstract Builder setBackendWorkerToken(String backendWorkerToken);
 
-    public abstract Builder setDirectEndpoint(WindmillServiceAddress value);
+    abstract Builder setDirectEndpoint(WindmillServiceAddress value);
 
-    public abstract Builder setStub(CloudWindmillServiceV1Alpha1Stub stub);
+    public abstract Builder setStubFactory(Supplier<CloudWindmillServiceV1Alpha1Stub> stubFactory);
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillConnection.java
user: login: m-trieu
body: done
created_at: 2025-04-08T18:29:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: right away

does autosizing allow specifying a minimum? is the initial a minimum?
created_at: 2025-04-09T15:08:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,9 +93,14 @@ public static ChannelCache create(
             new ThreadFactoryBuilder().setNameFormat("GrpcChannelCloser").build()));
   }
 
+  public static ChannelCache create(
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: scwhittle
body: maybe better to remove this and update callers to ignore settings?  then we can see at that call-site easier that they are present but ignored
created_at: 2025-04-09T15:31:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: how about a different constant used here and below for onready threshold. Can set them to the same value for now but less brittle to change.
created_at: 2025-04-09T15:34:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: seems like the DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS aren't being used if not passed here.

Seems like we should pass in and use in remoteChannel when configuring.  That would let us play around with the settings for non-direct jobs also.
created_at: 2025-04-09T15:35:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultDirectpathFlowControlSettings() {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: just make the constants public and remove the method? 
created_at: 2025-04-09T15:36:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultDirectpathFlowControlSettings() {
+    return DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultCloudpathFlowControlSettings() {
+    return DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  private static ManagedChannel remoteDirectChannel(
+      AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
+                    // Ports are required for direct channels.
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
+                    new AltsChannelCredentials.Builder().build())
+                .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
+            windmillServiceRpcChannelTimeoutSec);
+    int flowControlWindowSizeBytes =
+        Math.max(WINDMILL_MIN_FLOW_CONTROL_WINDOW, flowControlSettings.getFlowControlWindowBytes());
+    return flowControlSettings.getEnableAutoFlowControl()
+        ? channelBuilder.initialFlowControlWindow(flowControlWindowSizeBytes).build()
+        : channelBuilder.flowControlWindow(flowControlWindowSizeBytes).build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      HostAndPort endpoint, int windmillServiceRpcChannelTimeoutSec) {
+    return withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                endpoint.getHost(),
+                endpoint.hasPort()
+                    ? endpoint.getPort()
+                    : WindmillEndpoints.DEFAULT_WINDMILL_SERVICE_PORT),
+            windmillServiceRpcChannelTimeoutSec)
+        .negotiationType(NegotiationType.TLS)
+        .sslContext(dataflowGrpcSslContext(endpoint))
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: should we be doing this for remoteDirectChannel?
created_at: 2025-04-09T15:39:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultDirectpathFlowControlSettings() {
+    return DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultCloudpathFlowControlSettings() {
+    return DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  private static ManagedChannel remoteDirectChannel(
+      AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
+                    // Ports are required for direct channels.
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
+                    new AltsChannelCredentials.Builder().build())
+                .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
+            windmillServiceRpcChannelTimeoutSec);
+    int flowControlWindowSizeBytes =
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: probably have a helper to update based upon flow control settings to share with remote channel.
created_at: 2025-04-09T15:40:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: scwhittle
body: remove this from the default until we try it out?
created_at: 2025-04-09T15:41:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,10 +39,12 @@ public WindmillStubFactoryFactoryImpl(DataflowWorkerHarnessOptions workerOptions
   @Override
   public WindmillStubFactory makeWindmillStubFactory(boolean useIsolatedChannels) {
     Function<WindmillServiceAddress, ManagedChannel> channelFactory =
-        serviceAddress -> remoteChannel(serviceAddress, windmillServiceRpcChannelAliveTimeoutSec);
+        serviceAddress ->
+            remoteChannel(
+                serviceAddress.getServiceAddress(), windmillServiceRpcChannelAliveTimeoutSec);
     ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
+            (ignored, serviceAddress) ->
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
user: login: scwhittle
body: how about ignoredFlowControlSettings (or don't ignore it if we change remoteChannel to use it)
created_at: 2025-04-09T15:42:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -132,12 +136,80 @@ public void testClear() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (a, b) -> newChannel(channelName), notifyWhenChannelClosed::countDown);
     WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
     ManagedChannel cachedChannel = cache.get(someAddress);
     cache.clear();
     notifyWhenChannelClosed.await();
     assertTrue(cache.isEmpty());
     assertTrue(cachedChannel.isShutdown());
   }
+
+  @Test
+  public void testConsumeFlowControlSettings() throws InterruptedException {
+    String channelName = "channel";
+    CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    cache =
+        ChannelCache.forTesting(
+            (a, b) -> {
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: scwhittle
body: seems like you should keep track of what the settings passed in are, to make sure the plumbing is right. for example could add the settings to some concurrentqueue and then validate below
created_at: 2025-04-09T15:44:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -132,12 +136,80 @@ public void testClear() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (a, b) -> newChannel(channelName), notifyWhenChannelClosed::countDown);
     WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
     ManagedChannel cachedChannel = cache.get(someAddress);
     cache.clear();
     notifyWhenChannelClosed.await();
     assertTrue(cache.isEmpty());
     assertTrue(cachedChannel.isShutdown());
   }
+
+  @Test
+  public void testConsumeFlowControlSettings() throws InterruptedException {
+    String channelName = "channel";
+    CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    cache =
+        ChannelCache.forTesting(
+            (a, b) -> {
+              ManagedChannel channel = newChannel(channelName);
+              newChannelsCreated.incrementAndGet();
+              return channel;
+            },
+            notifyWhenChannelClosed::countDown);
+    WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
+    when(someAddress.getKind())
+        .thenReturn(WindmillServiceAddress.Kind.AUTHENTICATED_GCP_SERVICE_ADDRESS);
+    ManagedChannel cachedChannel = cache.get(someAddress);
+    cache.consumeFlowControlSettings(
+        UserWorkerGrpcFlowControlSettings.newBuilder()
+            .setEnableAutoFlowControl(true)
+            .setOnReadyThresholdBytes(1)
+            .setFlowControlWindowBytes(1)
+            .build());
+    ManagedChannel reloadedChannel = cache.get(someAddress);
+    notifyWhenChannelClosed.await();
+    assertThat(cachedChannel).isNotSameInstanceAs(reloadedChannel);
+    assertTrue(cachedChannel.isShutdown());
+    assertFalse(reloadedChannel.isShutdown());
+    assertThat(newChannelsCreated.get()).isEqualTo(2);
+    assertThat(cache.get(someAddress)).isSameInstanceAs(reloadedChannel);
+  }
+
+  @Test
+  public void testConsumeFlowControlSettings_sameFlowControlSettings() {
+    String channelName = "channel";
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    UserWorkerGrpcFlowControlSettings flowControlSettings =
+        UserWorkerGrpcFlowControlSettings.newBuilder()
+            .setEnableAutoFlowControl(true)
+            .setOnReadyThresholdBytes(1)
+            .setFlowControlWindowBytes(1)
+            .build();
+    cache =
+        ChannelCache.forTesting(
+            (a, b) -> {
+              ManagedChannel channel = newChannel(channelName);
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: scwhittle
body: ditto
created_at: 2025-04-09T15:44:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,7 +49,7 @@ public class ChannelCacheTest {
 
   private static ChannelCache newCache(
       Function<WindmillServiceAddress, ManagedChannel> channelFactory) {
-    return ChannelCache.forTesting(channelFactory, () -> {});
+    return ChannelCache.forTesting((ignored, address) -> channelFactory.apply(address), () -> {});
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: scwhittle
body: ignoredFlowControlSettings
created_at: 2025-04-09T15:45:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,7 +112,7 @@ public void testRemoveAndClose() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (a, b) -> newChannel(channelName), notifyWhenChannelClosed::countDown);
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: scwhittle
body: replace a,b with ignoredFlowControlSettings, ignoredAddress throughout

I'd rather have test be a little wordier than confusing
created_at: 2025-04-09T15:45:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -621,18 +623,45 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            (currentFlowControlSettings, serviceAddress) -> {
+              // IsolationChannel will create and manage separate RPC channels to the same
+              // serviceAddress.
+              return IsolationChannel.create(
+                  () ->
+                      remoteChannel(
+                          serviceAddress,
+                          workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec(),
+                          currentFlowControlSettings),
+                  currentFlowControlSettings.getOnReadyThresholdBytes());
+            });
+    channelCache.consumeFlowControlSettings(resolveInitialFlowControlSettings(configFetcher));
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: scwhittle
body: I think this line and the resolve method in this file can be removed, the cache internally can resolve unset or default settings appropriately. the fetcher observer should notify if there are overridden settings
created_at: 2025-04-09T15:48:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: done

https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html#initialFlowControlWindow(int)
i don't think so, just the initial window size.  the framework may resize it.
created_at: 2025-04-09T19:05:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,9 +93,14 @@ public static ChannelCache create(
             new ThreadFactoryBuilder().setNameFormat("GrpcChannelCloser").build()));
   }
 
+  public static ChannelCache create(
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: m-trieu
body: done
created_at: 2025-04-09T22:55:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: done
created_at: 2025-04-09T22:55:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: done
created_at: 2025-04-09T22:56:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: done
created_at: 2025-04-09T22:56:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultDirectpathFlowControlSettings() {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: done
created_at: 2025-04-09T23:44:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultDirectpathFlowControlSettings() {
+    return DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultCloudpathFlowControlSettings() {
+    return DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  private static ManagedChannel remoteDirectChannel(
+      AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
+                    // Ports are required for direct channels.
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
+                    new AltsChannelCredentials.Builder().build())
+                .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
+            windmillServiceRpcChannelTimeoutSec);
+    int flowControlWindowSizeBytes =
+        Math.max(WINDMILL_MIN_FLOW_CONTROL_WINDOW, flowControlSettings.getFlowControlWindowBytes());
+    return flowControlSettings.getEnableAutoFlowControl()
+        ? channelBuilder.initialFlowControlWindow(flowControlWindowSizeBytes).build()
+        : channelBuilder.flowControlWindow(flowControlWindowSizeBytes).build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      HostAndPort endpoint, int windmillServiceRpcChannelTimeoutSec) {
+    return withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                endpoint.getHost(),
+                endpoint.hasPort()
+                    ? endpoint.getPort()
+                    : WindmillEndpoints.DEFAULT_WINDMILL_SERVICE_PORT),
+            windmillServiceRpcChannelTimeoutSec)
+        .negotiationType(NegotiationType.TLS)
+        .sslContext(dataflowGrpcSslContext(endpoint))
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: i don't think so, directpath uses ALTS which doesn't need that.  I have not seen an issue pop up in testing
created_at: 2025-04-09T23:46:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import java.util.concurrent.TimeUnit;
+import javax.net.ssl.SSLException;
+import org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerGrpcFlowControlSettings;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillEndpoints;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress.AuthenticatedGcpServiceAddress;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.alts.AltsChannelCredentials;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p69p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p69p0.io.netty.handler.ssl.SslContext;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+
+/** Utility class used to create different RPC Channels. */
+public final class WindmillChannels {
+  public static final String LOCALHOST = "localhost";
+  private static final int MAX_REMOTE_TRACE_EVENTS = 100;
+  // 1MiB.
+  private static final int MAX_INBOUND_METADATA_SIZE_BYTES = 1024 * 1024;
+  // 10MiB. Roughly 2x max message size.
+  private static final int WINDMILL_MIN_FLOW_CONTROL_WINDOW =
+      NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 10;
+
+  // User a bigger flow control window and onready threshold for directpath to prevent churn when
+  // gRPC is trying to flush gRPCs over the wire. If onReadyThreshold and flowControlWindowBytes are
+  // too low, it was observed in testing that the gRPC channel can get stuck in a "not-ready" state
+  // until stream deadline.
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          // 100MiB.
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // 100MiB.
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW * 10)
+          // Prevent gRPC from automatically resizing the window. If we have things to send/receive
+          // from Windmill we want to do it right way. There are internal pushback mechanisms in the
+          // user worker and Windmill that attempt to guard the process from OOMing (i.e
+          // MemoryMonitor.java).
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private static final UserWorkerGrpcFlowControlSettings DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS =
+      UserWorkerGrpcFlowControlSettings.newBuilder()
+          .setFlowControlWindowBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setOnReadyThresholdBytes(WINDMILL_MIN_FLOW_CONTROL_WINDOW)
+          .setEnableAutoFlowControl(false)
+          .build();
+
+  private WindmillChannels() {}
+
+  public static ManagedChannel inProcessChannel(String channelName) {
+    return InProcessChannelBuilder.forName(channelName).directExecutor().build();
+  }
+
+  public static Channel localhostChannel(int port) {
+    return NettyChannelBuilder.forAddress(LOCALHOST, port)
+        .maxInboundMessageSize(Integer.MAX_VALUE)
+        .negotiationType(NegotiationType.PLAINTEXT)
+        .build();
+  }
+
+  public static ManagedChannel remoteChannel(
+      WindmillServiceAddress windmillServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    switch (windmillServiceAddress.getKind()) {
+      case GCP_SERVICE_ADDRESS:
+        return remoteChannel(
+            windmillServiceAddress.gcpServiceAddress(), windmillServiceRpcChannelTimeoutSec);
+      case AUTHENTICATED_GCP_SERVICE_ADDRESS:
+        return remoteDirectChannel(
+            windmillServiceAddress.authenticatedGcpServiceAddress(),
+            windmillServiceRpcChannelTimeoutSec,
+            flowControlSettings);
+      default:
+        throw new UnsupportedOperationException(
+            "Only GCP_SERVICE_ADDRESS and AUTHENTICATED_GCP_SERVICE_ADDRESS are supported"
+                + " WindmillServiceAddresses.");
+    }
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultDirectpathFlowControlSettings() {
+    return DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  public static UserWorkerGrpcFlowControlSettings getDefaultCloudpathFlowControlSettings() {
+    return DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS;
+  }
+
+  private static ManagedChannel remoteDirectChannel(
+      AuthenticatedGcpServiceAddress authenticatedGcpServiceAddress,
+      int windmillServiceRpcChannelTimeoutSec,
+      UserWorkerGrpcFlowControlSettings flowControlSettings) {
+    NettyChannelBuilder channelBuilder =
+        withDefaultChannelOptions(
+            NettyChannelBuilder.forAddress(
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getHost(),
+                    // Ports are required for direct channels.
+                    authenticatedGcpServiceAddress.gcpServiceAddress().getPort(),
+                    new AltsChannelCredentials.Builder().build())
+                .overrideAuthority(authenticatedGcpServiceAddress.authenticatingService()),
+            windmillServiceRpcChannelTimeoutSec);
+    int flowControlWindowSizeBytes =
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillChannels.java
user: login: m-trieu
body: done
created_at: 2025-04-09T23:55:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,10 +39,12 @@ public WindmillStubFactoryFactoryImpl(DataflowWorkerHarnessOptions workerOptions
   @Override
   public WindmillStubFactory makeWindmillStubFactory(boolean useIsolatedChannels) {
     Function<WindmillServiceAddress, ManagedChannel> channelFactory =
-        serviceAddress -> remoteChannel(serviceAddress, windmillServiceRpcChannelAliveTimeoutSec);
+        serviceAddress ->
+            remoteChannel(
+                serviceAddress.getServiceAddress(), windmillServiceRpcChannelAliveTimeoutSec);
     ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
+            (ignored, serviceAddress) ->
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
user: login: m-trieu
body: updated, using in `remoteChannel`
created_at: 2025-04-09T23:56:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -621,18 +623,45 @@ private static void validateWorkerOptions(DataflowWorkerHarnessOptions options)
   }
 
   private static ChannelCachingStubFactory createFanOutStubFactory(
-      DataflowWorkerHarnessOptions workerOptions) {
-    return ChannelCachingRemoteStubFactory.create(
-        workerOptions.getGcpCredential(),
+      DataflowWorkerHarnessOptions workerOptions, ComputationConfig.Fetcher configFetcher) {
+    ChannelCache channelCache =
         ChannelCache.create(
-            serviceAddress ->
-                // IsolationChannel will create and manage separate RPC channels to the same
-                // serviceAddress.
-                IsolationChannel.create(
-                    () ->
-                        remoteChannel(
-                            serviceAddress,
-                            workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec()))));
+            (currentFlowControlSettings, serviceAddress) -> {
+              // IsolationChannel will create and manage separate RPC channels to the same
+              // serviceAddress.
+              return IsolationChannel.create(
+                  () ->
+                      remoteChannel(
+                          serviceAddress,
+                          workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec(),
+                          currentFlowControlSettings),
+                  currentFlowControlSettings.getOnReadyThresholdBytes());
+            });
+    channelCache.consumeFlowControlSettings(resolveInitialFlowControlSettings(configFetcher));
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
user: login: m-trieu
body: done
created_at: 2025-04-10T07:32:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -108,7 +112,7 @@ public void testRemoveAndClose() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (a, b) -> newChannel(channelName), notifyWhenChannelClosed::countDown);
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: m-trieu
body: done
created_at: 2025-04-10T07:35:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,7 +49,7 @@ public class ChannelCacheTest {
 
   private static ChannelCache newCache(
       Function<WindmillServiceAddress, ManagedChannel> channelFactory) {
-    return ChannelCache.forTesting(channelFactory, () -> {});
+    return ChannelCache.forTesting((ignored, address) -> channelFactory.apply(address), () -> {});
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: m-trieu
body: done
created_at: 2025-04-10T07:35:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -132,12 +136,80 @@ public void testClear() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (a, b) -> newChannel(channelName), notifyWhenChannelClosed::countDown);
     WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
     ManagedChannel cachedChannel = cache.get(someAddress);
     cache.clear();
     notifyWhenChannelClosed.await();
     assertTrue(cache.isEmpty());
     assertTrue(cachedChannel.isShutdown());
   }
+
+  @Test
+  public void testConsumeFlowControlSettings() throws InterruptedException {
+    String channelName = "channel";
+    CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    cache =
+        ChannelCache.forTesting(
+            (a, b) -> {
+              ManagedChannel channel = newChannel(channelName);
+              newChannelsCreated.incrementAndGet();
+              return channel;
+            },
+            notifyWhenChannelClosed::countDown);
+    WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
+    when(someAddress.getKind())
+        .thenReturn(WindmillServiceAddress.Kind.AUTHENTICATED_GCP_SERVICE_ADDRESS);
+    ManagedChannel cachedChannel = cache.get(someAddress);
+    cache.consumeFlowControlSettings(
+        UserWorkerGrpcFlowControlSettings.newBuilder()
+            .setEnableAutoFlowControl(true)
+            .setOnReadyThresholdBytes(1)
+            .setFlowControlWindowBytes(1)
+            .build());
+    ManagedChannel reloadedChannel = cache.get(someAddress);
+    notifyWhenChannelClosed.await();
+    assertThat(cachedChannel).isNotSameInstanceAs(reloadedChannel);
+    assertTrue(cachedChannel.isShutdown());
+    assertFalse(reloadedChannel.isShutdown());
+    assertThat(newChannelsCreated.get()).isEqualTo(2);
+    assertThat(cache.get(someAddress)).isSameInstanceAs(reloadedChannel);
+  }
+
+  @Test
+  public void testConsumeFlowControlSettings_sameFlowControlSettings() {
+    String channelName = "channel";
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    UserWorkerGrpcFlowControlSettings flowControlSettings =
+        UserWorkerGrpcFlowControlSettings.newBuilder()
+            .setEnableAutoFlowControl(true)
+            .setOnReadyThresholdBytes(1)
+            .setFlowControlWindowBytes(1)
+            .build();
+    cache =
+        ChannelCache.forTesting(
+            (a, b) -> {
+              ManagedChannel channel = newChannel(channelName);
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: m-trieu
body: done
created_at: 2025-04-10T07:35:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -132,12 +136,80 @@ public void testClear() throws InterruptedException {
     CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
     cache =
         ChannelCache.forTesting(
-            ignored -> newChannel(channelName), notifyWhenChannelClosed::countDown);
+            (a, b) -> newChannel(channelName), notifyWhenChannelClosed::countDown);
     WindmillServiceAddress someAddress = mock(WindmillServiceAddress.class);
     ManagedChannel cachedChannel = cache.get(someAddress);
     cache.clear();
     notifyWhenChannelClosed.await();
     assertTrue(cache.isEmpty());
     assertTrue(cachedChannel.isShutdown());
   }
+
+  @Test
+  public void testConsumeFlowControlSettings() throws InterruptedException {
+    String channelName = "channel";
+    CountDownLatch notifyWhenChannelClosed = new CountDownLatch(1);
+    AtomicInteger newChannelsCreated = new AtomicInteger();
+    cache =
+        ChannelCache.forTesting(
+            (a, b) -> {
path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
user: login: m-trieu
body: done, we only call once so using AtomicRef
created_at: 2025-04-10T07:40:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -169,8 +160,16 @@ boolean isEmpty() {
   public void appendSummaryHtml(PrintWriter writer) {
     synchronized (this) {
       if (currentFlowControlSettings != null) {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: scwhittle
body: how about making resolveFlowControlSettings a private member of this class so this method and the cache loader can share it.
created_at: 2025-04-10T13:42:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -169,8 +160,16 @@ boolean isEmpty() {
   public void appendSummaryHtml(PrintWriter writer) {
     synchronized (this) {
       if (currentFlowControlSettings != null) {
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: m-trieu
body: sgtm, im going to modify the code to remove `WindmillStubFactoryFactory` after this PR is submitted now that we have the `ChannelCache` that can reload based on job settings updates.
created_at: 2025-04-10T21:30:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -158,27 +150,34 @@ boolean isEmpty() {
 
   @Override
   public void appendSummaryHtml(PrintWriter writer) {
-    synchronized (this) {
-      if (currentFlowControlSettings != null) {
-        writer.format("Current gRPC flow control settings: [%s]", currentFlowControlSettings);
-      } else {
-        writer.format(
-            "Cloudpath gRPC flow control settings: [%s]",
-            WindmillChannels.DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS);
-        writer.write("<br>");
-        writer.format(
-            "Directpath gRPC flow control settings: [%s]",
-            WindmillChannels.DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS);
-        writer.write("<br>");
-      }
-    }
+    writer.format(
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: damccorm
body: Won't this now just write the same string twice if `currentFlowControlSettings` is not null?
created_at: 2025-04-15T14:59:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -158,27 +150,34 @@ boolean isEmpty() {
 
   @Override
   public void appendSummaryHtml(PrintWriter writer) {
-    synchronized (this) {
-      if (currentFlowControlSettings != null) {
-        writer.format("Current gRPC flow control settings: [%s]", currentFlowControlSettings);
-      } else {
-        writer.format(
-            "Cloudpath gRPC flow control settings: [%s]",
-            WindmillChannels.DEFAULT_CLOUDPATH_FLOW_CONTROL_SETTINGS);
-        writer.write("<br>");
-        writer.format(
-            "Directpath gRPC flow control settings: [%s]",
-            WindmillChannels.DEFAULT_DIRECTPATH_FLOW_CONTROL_SETTINGS);
-        writer.write("<br>");
-      }
-    }
+    writer.format(
path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
user: login: m-trieu
body: done
created_at: 2025-04-15T21:35:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: m-trieu
created_at: 2025-04-03T22:37:45Z
body: R: @scwhittle 
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-03T22:39:01Z
body: Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
author_association: CONTRIBUTOR
user: login: m-trieu
created_at: 2025-04-04T00:40:03Z
body: @scwhittle this may lead to some retries in Windmill after we break the streams but it should not happen often as it is controlled by explicit updates to the flow control settings pushed from the Windmill backend.

maintain default settings for Cloudpath
author_association: CONTRIBUTOR
user: login: m-trieu
created_at: 2025-04-10T07:53:51Z
body: back to you @scwhittle thanks!
author_association: CONTRIBUTOR
user: login: m-trieu
created_at: 2025-04-14T17:18:50Z
body: assign set of reviewers
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-14T17:20:18Z
body: Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:

R: @damccorm added as fallback since no labels match configuration

Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-04-14T17:21:03Z
body: ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/34539?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 56.42%. Comparing base [(`cad045b`)](https://app.codecov.io/gh/apache/beam/commit/cad045b41fc83ad23dd9ea7944ba7b3e4f71c5a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`6ff995b`)](https://app.codecov.io/gh/apache/beam/commit/6ff995bf2e80322631423df68f587d6ee6223675?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 95 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master   #34539      +/-   ##
============================================
- Coverage     56.42%   56.42%   -0.01%     
  Complexity     3285     3285              
============================================
  Files          1177     1177              
  Lines        180098   180097       -1     
  Branches       3398     3398              
============================================
- Hits         101620   101619       -1     
  Misses        75216    75216              
  Partials       3262     3262              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/34539?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: m-trieu
created_at: 2025-04-14T17:24:48Z
body: hi @damccorm! This is ready for a merge (see https://github.com/apache/beam/pull/34539#pullrequestreview-2756874459), Sam is just OOO.

Thank you!
author_association: CONTRIBUTOR