Repository: alibaba/fluss
Pull request #845: [server] Add retriable authentication exception.
Author: login: loserwang1024
Created: 2025-05-09T07:56:35Z | Merged: 2025-05-19T02:37:42Z

<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*

## Contribution Checklist

  - Make sure that the pull request corresponds to a [GitHub issue](https://github.com/alibaba/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.

  - Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.

  - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.

  - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.

  - Each pull request should address only one issue, not mix up code from multiple issues.


**(The sections below can be removed for hotfixes or typos)**
-->

### Purpose

<!-- Linking this pull request to the issue -->
Linked issue: close #844 

<!-- What is the purpose of the change -->

### Brief change log

<!-- Please describe the changes made in this pull request and explain how they address the issue -->

### Tests

com.alibaba.fluss.rpc.netty.authenticate.AuthenticationTest#testRetirableAuthenticateException

### API and Format

<!-- Does this change affect API or storage format -->

### Documentation

<!-- Does this change introduce a new feature -->


Linked issue #844: Add retriable authentication exception
### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.


### Motivation

Currently,  authentication exception is not a retriable exception. However, sometimes, the exception doesn't occur because credential(eg, username and password) is wrong but some retriable exception. For example, throttling or network jitter with outside authentication system.


### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Diff:
diff --git a/LICENSE b/LICENSE
index 8c872ec776..8ba480bac8 100644
--- a/LICENSE
+++ b/LICENSE
@@ -311,6 +311,7 @@ Apache Kafka
 ./fluss-common/src/main/java/com/alibaba/fluss/utils/crc/PureJavaCrc32C.java
 ./fluss-common/src/main/java/com/alibaba/fluss/utils/log/ByteBufferUnmapper.java
 ./fluss-common/src/main/java/com/alibaba/fluss/utils/log/FairBucketStatusMap.java
+./fluss-common/src/main/java/com/alibaba/fluss/utils/ExponentialBackoff.java
 ./fluss-server/src/main/java/com/alibaba/fluss/server/coordinator/statemachine/ReplicaStateMachine.java
 ./fluss-server/src/main/java/com/alibaba/fluss/server/coordinator/statemachine/TableBucketStateMachine.java
 ./fluss-server/src/main/java/com/alibaba/fluss/server/log/AbstractIndex.java
diff --git a/fluss-common/src/main/java/com/alibaba/fluss/exception/RetriableAuthenticationException.java b/fluss-common/src/main/java/com/alibaba/fluss/exception/RetriableAuthenticationException.java
new file mode 100644
index 0000000000..b689e8186f
--- /dev/null
+++ b/fluss-common/src/main/java/com/alibaba/fluss/exception/RetriableAuthenticationException.java
@@ -0,0 +1,28 @@
+/*
+ *  Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package com.alibaba.fluss.exception;
+
+/**
+ * This exception is thrown if authentication fails with a retriable error.
+ *
+ * @since 0.7
+ */
+public class RetriableAuthenticationException extends AuthenticationException {
+    public RetriableAuthenticationException(String message) {
+        super(message);
+    }
+}
diff --git a/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ClientAuthenticator.java b/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ClientAuthenticator.java
index d7b6f333df..6316fb7065 100644
--- a/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ClientAuthenticator.java
+++ b/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ClientAuthenticator.java
@@ -28,6 +28,9 @@ public interface ClientAuthenticator {
     /** The protocol name of the authenticator, which will send in the AuthenticateRequest. */
     String protocol();
 
+    /** Initialize the authenticator. */
+    default void initialize(AuthenticateContext context) {}
+
     /**
      * * Generates the initial token or calculates a token based on the server's challenge, then
      * sends it back to the server. This method sets the client authentication status as complete if
@@ -75,4 +78,7 @@ public interface ClientAuthenticator {
 
     /** Checks if the authentication from client side is completed. */
     boolean isCompleted();
+
+    /** The context of the authentication process. */
+    interface AuthenticateContext {}
 }
diff --git a/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ServerAuthenticator.java b/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ServerAuthenticator.java
index 0b7a195d33..3098b39468 100644
--- a/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ServerAuthenticator.java
+++ b/fluss-common/src/main/java/com/alibaba/fluss/security/auth/ServerAuthenticator.java
@@ -30,6 +30,9 @@ public interface ServerAuthenticator {
 
     String protocol();
 
+    /** Initialize the authenticator. */
+    default void initialize(AuthenticateContext context) {}
+
     /**
      * * Generates the challenge based on the client's token, then sends it back to the client. This
      * method sets the server authentication status as complete if the authentication succeeds.
@@ -79,4 +82,7 @@ public interface ServerAuthenticator {
      * complete).
      */
     FlussPrincipal createPrincipal();
+
+    /** The context of the authentication process. */
+    interface AuthenticateContext {}
 }
diff --git a/fluss-common/src/main/java/com/alibaba/fluss/utils/ExponentialBackoff.java b/fluss-common/src/main/java/com/alibaba/fluss/utils/ExponentialBackoff.java
new file mode 100644
index 0000000000..3d5f440998
--- /dev/null
+++ b/fluss-common/src/main/java/com/alibaba/fluss/utils/ExponentialBackoff.java
@@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.utils;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.util.concurrent.ThreadLocalRandom;
+
+/* This file is based on source code of Apache Kafka Project (https://kafka.apache.org/), licensed by the Apache
+ * Software Foundation (ASF) under the Apache License, Version 2.0. See the NOTICE file distributed with this work for
+ * additional information regarding copyright ownership. */
+
+/**
+ * A utility class for computing exponential backoff values, commonly used for retry logic,
+ * reconnection attempts, and timeout management.
+ *
+ * <p>The backoff interval increases exponentially with each attempt, following the formula:
+ *
+ * <pre>
+ * Backoff(attempts) = random(1 - jitter, 1 + jitter) * initialInterval * multiplier^attempts
+ * </pre>
+ *
+ * <p>If {@code maxInterval} is less than {@code initialInterval}, a constant backoff of {@code
+ * maxInterval} will be applied. The jitter factor ensures randomness to avoid thundering herd
+ * problems, but will never cause the result to exceed the configured maximum interval.
+ *
+ * <p>Instances of this class are thread-safe and can be shared across multiple threads.
+ */
+@ThreadSafe
+public class ExponentialBackoff {
+    private final long initialInterval;
+    private final int multiplier;
+    private final long maxInterval;
+    private final double jitter;
+    private final double expMax;
+
+    public ExponentialBackoff(
+            long initialInterval, int multiplier, long maxInterval, double jitter) {
+        this.initialInterval = Math.min(maxInterval, initialInterval);
+        this.multiplier = multiplier;
+        this.maxInterval = maxInterval;
+        this.jitter = jitter;
+        this.expMax =
+                maxInterval > initialInterval
+                        ? Math.log(maxInterval / (double) Math.max(initialInterval, 1))
+                                / Math.log(multiplier)
+                        : 0;
+    }
+
+    public long backoff(long attempts) {
+        if (expMax == 0) {
+            return initialInterval;
+        }
+        double exp = Math.min(attempts, this.expMax);
+        double term = initialInterval * Math.pow(multiplier, exp);
+        double randomFactor =
+                jitter < Double.MIN_NORMAL
+                        ? 1.0
+                        : ThreadLocalRandom.current().nextDouble(1 - jitter, 1 + jitter);
+        long backoffValue = (long) (randomFactor * term);
+        return Math.min(backoffValue, maxInterval);
+    }
+
+    @Override
+    public String toString() {
+        return "ExponentialBackoff{"
+                + "multiplier="
+                + multiplier
+                + ", expMax="
+                + expMax
+                + ", initialInterval="
+                + initialInterval
+                + ", jitter="
+                + jitter
+                + '}';
+    }
+}
diff --git a/fluss-common/src/test/java/com/alibaba/fluss/utils/ExponentialBackoffTest.java b/fluss-common/src/test/java/com/alibaba/fluss/utils/ExponentialBackoffTest.java
new file mode 100644
index 0000000000..4cbaf88e38
--- /dev/null
+++ b/fluss-common/src/test/java/com/alibaba/fluss/utils/ExponentialBackoffTest.java
@@ -0,0 +1,58 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.utils;
+
+import org.assertj.core.data.Percentage;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Test for {@link ExponentialBackoff}. */
+public class ExponentialBackoffTest {
+    @Test
+    public void testExponentialBackoff() {
+        long initialValue = 100;
+        int ratio = 2;
+        long backoffMax = 2000;
+        double jitter = 0.2;
+        ExponentialBackoff exponentialBackoff =
+                new ExponentialBackoff(initialValue, ratio, backoffMax, jitter);
+
+        for (int i = 0; i <= 100; i++) {
+            for (int attempts = 0; attempts <= 10; attempts++) {
+                if (attempts <= 4) {
+                    assertThat(1.0 * exponentialBackoff.backoff(attempts))
+                            .isCloseTo(
+                                    initialValue * Math.pow(ratio, attempts),
+                                    Percentage.withPercentage(jitter * 100));
+                } else {
+                    assertThat(exponentialBackoff.backoff(attempts) <= backoffMax * (1 + jitter))
+                            .isTrue();
+                }
+            }
+        }
+    }
+
+    @Test
+    public void testExponentialBackoffWithoutJitter() {
+        ExponentialBackoff exponentialBackoff = new ExponentialBackoff(100, 2, 400, 0.0);
+        assertThat(exponentialBackoff.backoff(0)).isEqualTo(100);
+        assertThat(exponentialBackoff.backoff(1)).isEqualTo(200);
+        assertThat(exponentialBackoff.backoff(2)).isEqualTo(400);
+        assertThat(exponentialBackoff.backoff(3)).isEqualTo(400);
+    }
+}
diff --git a/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java b/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java
index 42d2e2980f..68be9ea8f6 100644
--- a/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java
+++ b/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java
@@ -20,6 +20,7 @@
 import com.alibaba.fluss.exception.DisconnectException;
 import com.alibaba.fluss.exception.FlussRuntimeException;
 import com.alibaba.fluss.exception.NetworkException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
 import com.alibaba.fluss.rpc.messages.ApiMessage;
 import com.alibaba.fluss.rpc.messages.ApiVersionsRequest;
 import com.alibaba.fluss.rpc.messages.ApiVersionsResponse;
@@ -38,6 +39,7 @@
 import com.alibaba.fluss.shaded.netty4.io.netty.channel.Channel;
 import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelFuture;
 import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelFutureListener;
+import com.alibaba.fluss.utils.ExponentialBackoff;
 import com.alibaba.fluss.utils.MapUtils;
 
 import org.slf4j.Logger;
@@ -52,6 +54,7 @@
 import java.util.ArrayDeque;
 import java.util.Map;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
 import java.util.function.Consumer;
 
 /** Connection to a Netty server used by the {@link NettyClient}. */
@@ -66,6 +69,7 @@ final class ServerConnection {
     private final CompletableFuture<Void> closeFuture = new CompletableFuture<>();
     private final ConnectionMetricGroup connectionMetricGroup;
     private final ClientAuthenticator authenticator;
+    private final ExponentialBackoff backoff;
 
     private final Object lock = new Object();
 
@@ -87,6 +91,9 @@ final class ServerConnection {
     @GuardedBy("lock")
     private ServerApiVersions serverApiVersions;
 
+    @GuardedBy("lock")
+    private int retryAuthCount = 0;
+
     ServerConnection(
             Bootstrap bootstrap,
             ServerNode node,
@@ -99,6 +106,7 @@ final class ServerConnection {
                 .connect(node.host(), node.port())
                 .addListener((ChannelFutureListener) this::establishConnection);
         this.authenticator = authenticator;
+        this.backoff = new ExponentialBackoff(100L, 2, 5000L, 0.2);
     }
 
     public ServerNode getServerNode() {
@@ -366,13 +374,18 @@ private void handleApiVersionsResponse(ApiMessage response, Throwable cause) {
         synchronized (lock) {
             serverApiVersions =
                     new ServerApiVersions(((ApiVersionsResponse) response).getApiVersionsList());
-            LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
             // send initial token
-            sendAuthenticate(new byte[0]);
+            sendInitialToken();
         }
     }
 
-    private void sendAuthenticate(byte[] challenge) {
+    private void sendInitialToken() {
+        authenticator.initialize(new DefaultAuthenticateContext());
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        sendAuthenticateRequest(new byte[0]);
+    }
+
+    private void sendAuthenticateRequest(byte[] challenge) {
         try {
             if (!authenticator.isCompleted()) {
                 byte[] token = authenticator.authenticate(challenge);
@@ -403,19 +416,28 @@ private void sendAuthenticate(byte[] challenge) {
     }
 
     private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
-        if (cause != null) {
-            close(cause);
-            return;
-        }
-        if (!(response instanceof AuthenticateResponse)) {
-            close(new IllegalStateException("Unexpected response type " + response.getClass()));
-            return;
-        }
-
         synchronized (lock) {
+            if (cause != null) {
+                if (cause instanceof RetriableAuthenticationException) {
+                    LOG.warn("Authentication failed, retrying {} times", retryAuthCount, cause);
+                    channel.eventLoop()
+                            .schedule(
+                                    this::sendInitialToken,
+                                    backoff.backoff(retryAuthCount++),
+                                    TimeUnit.MILLISECONDS);
+                } else {
+                    close(cause);
+                }
+                return;
+            }
+            if (!(response instanceof AuthenticateResponse)) {
+                close(new IllegalStateException("Unexpected response type " + response.getClass()));
+                return;
+            }
+
             AuthenticateResponse authenticateResponse = (AuthenticateResponse) response;
             if (authenticateResponse.hasChallenge()) {
-                sendAuthenticate(((AuthenticateResponse) response).getChallenge());
+                sendAuthenticateRequest(((AuthenticateResponse) response).getChallenge());
             } else if (authenticator.isCompleted()) {
                 switchState(ConnectionState.READY);
             } else {
@@ -521,4 +543,7 @@ ByteBuf toByteBuf(ByteBufAllocator allocator) {
             return MessageCodec.encodeRequest(allocator, apiKey, apiVersion, requestId, request);
         }
     }
+
+    private static class DefaultAuthenticateContext
+            implements ClientAuthenticator.AuthenticateContext {}
 }
diff --git a/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/server/NettyServerHandler.java b/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/server/NettyServerHandler.java
index f0e8655ae3..428141edd2 100644
--- a/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/server/NettyServerHandler.java
+++ b/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/server/NettyServerHandler.java
@@ -19,6 +19,7 @@
 import com.alibaba.fluss.annotation.VisibleForTesting;
 import com.alibaba.fluss.exception.AuthenticationException;
 import com.alibaba.fluss.exception.NetworkException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
 import com.alibaba.fluss.record.send.Send;
 import com.alibaba.fluss.rpc.messages.ApiMessage;
 import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
@@ -162,6 +163,7 @@ public void channelActive(ChannelHandlerContext ctx) throws Exception {
         super.channelActive(ctx);
         this.ctx = ctx;
         this.remoteAddress = ctx.channel().remoteAddress();
+        authenticator.initialize(new DefaultAuthenticateContext());
         switchState(
                 authenticator.isCompleted()
                         ? ConnectionState.READY
@@ -316,14 +318,27 @@ private void handleAuthenticateRequest(
         }
 
         AuthenticateResponse authenticateResponse = new AuthenticateResponse();
-        if (!authenticator.isCompleted()) {
-            byte[] token = authenticateRequest.getToken();
-            byte[] challenge = authenticator.evaluateResponse(token);
-            if (!authenticator.isCompleted() && challenge != null) {
-                authenticateResponse.setChallenge(challenge);
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticateRequest.getToken();
+                byte[] challenge = authenticator.evaluateResponse(token);
+                if (challenge != null) {
+                    authenticateResponse.setChallenge(challenge);
+                }
+            }
+            future.complete(authenticateResponse);
+        } catch (AuthenticationException e) {
+            if (e instanceof RetriableAuthenticationException) {
+                LOG.warn(
+                        "Authentication from {} failed due to a retriable exception: {}. Reinitializing authenticator for subsequent retries.",
+                        ctx.channel().remoteAddress(),
+                        e.getMessage(),
+                        e);
+                authenticator.initialize(new DefaultAuthenticateContext());
             }
+
+            future.completeExceptionally(e);
         }
-        future.complete(authenticateResponse);
 
         if (authenticator.isCompleted()) {
             switchState(ConnectionState.READY);
@@ -349,4 +364,7 @@ public boolean isAuthenticating() {
             return this == AUTHENTICATING;
         }
     }
+
+    private static class DefaultAuthenticateContext
+            implements ServerAuthenticator.AuthenticateContext {}
 }
diff --git a/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/protocol/Errors.java b/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/protocol/Errors.java
index b83aa36b56..e5b2382518 100644
--- a/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/protocol/Errors.java
+++ b/fluss-rpc/src/main/java/com/alibaba/fluss/rpc/protocol/Errors.java
@@ -54,6 +54,7 @@
 import com.alibaba.fluss.exception.PartitionAlreadyExistsException;
 import com.alibaba.fluss.exception.PartitionNotExistException;
 import com.alibaba.fluss.exception.RecordTooLargeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
 import com.alibaba.fluss.exception.SchemaNotExistException;
 import com.alibaba.fluss.exception.SecurityDisabledException;
 import com.alibaba.fluss.exception.SecurityTokenException;
@@ -202,7 +203,11 @@ public enum Errors {
     BUCKET_MAX_NUM_EXCEPTION(
             49, "Exceed the maximum number of buckets", TooManyBucketsException::new),
     FENCED_TIERING_EPOCH_EXCEPTION(
-            50, "The tiering epoch is invalid.", FencedTieringEpochException::new);
+            50, "The tiering epoch is invalid.", FencedTieringEpochException::new),
+    RETRIABLE_AUTHENTICATE_EXCEPTION(
+            51,
+            "Authentication failed with retriable exception. ",
+            RetriableAuthenticationException::new);
 
     private static final Logger LOG = LoggerFactory.getLogger(Errors.class);
 
diff --git a/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/AuthenticationTest.java b/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/AuthenticationTest.java
index 536e1808af..5f5939cd63 100644
--- a/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/AuthenticationTest.java
+++ b/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/AuthenticationTest.java
@@ -122,6 +122,17 @@ void testNoChallengeBeforeClientComplete() throws Exception {
         }
     }
 
+    @Test
+    void testRetirableAuthenticateException() throws Exception {
+        Configuration clientConfig = new Configuration();
+        clientConfig.set(ConfigOptions.CLIENT_SECURITY_PROTOCOL, "mutual");
+        clientConfig.setString("client.security.mutual.error-type", "RETRIABLE_EXCEPTION");
+        try (NettyClient nettyClient =
+                new NettyClient(clientConfig, TestingClientMetricGroup.newInstance())) {
+            verifyGetTableNamesList(nettyClient, mutualAuthServerNode);
+        }
+    }
+
     @Test
     void testClientLackAuthenticateProtocol() throws Exception {
         Configuration clientConfig = new Configuration();
diff --git a/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/MutualAuthenticationPlugin.java b/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/MutualAuthenticationPlugin.java
index 4159d6a8af..b2fd526319 100644
--- a/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/MutualAuthenticationPlugin.java
+++ b/fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/authenticate/MutualAuthenticationPlugin.java
@@ -19,6 +19,7 @@
 import com.alibaba.fluss.config.ConfigOption;
 import com.alibaba.fluss.config.Configuration;
 import com.alibaba.fluss.exception.AuthenticationException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
 import com.alibaba.fluss.security.acl.FlussPrincipal;
 import com.alibaba.fluss.security.auth.ClientAuthenticationPlugin;
 import com.alibaba.fluss.security.auth.ClientAuthenticator;
@@ -57,8 +58,15 @@ public String authProtocol() {
     }
 
     private static class ClientAuthenticatorImpl implements ClientAuthenticator {
-        private boolean isCompleted = false;
-        private int initialSalt;
+        private enum Status {
+            SEND_CLIENT_FIRST_MESSAGE,
+            RECEIVE_SERVER_FIRST_MESSAGE,
+            RECEIVE_SERVER_FINAL_MESSAGE,
+            COMPLETED
+        }
+
+        private Status status;
+        Integer initialSalt;
         private final int errorType;
 
         public ClientAuthenticatorImpl(Configuration configuration) {
@@ -71,85 +79,115 @@ public String protocol() {
         }
 
         @Override
-        public byte[] authenticate(byte[] data) throws AuthenticationException {
-            if (isCompleted) {
-                return null;
-            }
-
-            // Initial token
-            if (data.length == 0) {
-                initialSalt = generateInitialSalt();
-                return String.valueOf(
-                                isError(
-                                                errorType,
-                                                ErrorType.SERVER_NO_CHALLENGE,
-                                                ErrorType.SERVER_ERROR_CHALLENGE)
-                                        ? errorType
-                                        : initialSalt)
-                        .getBytes();
-            }
+        public void initialize(AuthenticateContext context) {
+            this.status = Status.SEND_CLIENT_FIRST_MESSAGE;
+            this.initialSalt = null;
+        }
 
-            int challenge = parseToken(data);
-            if (challenge == initialSalt + 1) {
-                isCompleted = true;
-                return String.valueOf(
-                                isError(errorType, ErrorType.CLIENT_ERROR_SECOND_TOKEN)
-                                        ? errorType
-                                        : challenge + 1)
-                        .getBytes();
+        @Override
+        public byte[] authenticate(byte[] data) throws AuthenticationException {
+            switch (status) {
+                case SEND_CLIENT_FIRST_MESSAGE:
+                    initialSalt =
+                            isError(
+                                            errorType,
+                                            ErrorType.SERVER_NO_CHALLENGE,
+                                            ErrorType.SERVER_ERROR_CHALLENGE,
+                                            ErrorType.RETRIABLE_EXCEPTION)
+                                    ? errorType
+                                    : generateInitialSalt();
+                    status = Status.RECEIVE_SERVER_FIRST_MESSAGE;
+                    return String.valueOf(initialSalt).getBytes();
+                case RECEIVE_SERVER_FIRST_MESSAGE:
+                    int challenge = parseToken(data);
+                    if (challenge == initialSalt + 1) {
+                        status = Status.RECEIVE_SERVER_FINAL_MESSAGE;
+                        return String.valueOf(
+                                        isError(errorType, ErrorType.CLIENT_ERROR_SECOND_TOKEN)
+                                                ? errorType
+                                                : challenge + 1)
+                                .getBytes();
+                    } else {
+                        throw new AuthenticationException(
+                                "Invalid challenge value: expected "
+                                        + (initialSalt + 1)
+                                        + ", but got "
+                                        + challenge);
+                    }
+                case RECEIVE_SERVER_FINAL_MESSAGE:
+                    if (data.length == 0) {
+                        status = Status.COMPLETED;
+                        return null;
+                    } else {
+                        throw new AuthenticationException(
+                                "Invalid token value: expected empty token, but got "
+                                        + new String(data));
+                    }
+                default:
+                    return null;
             }
-
-            throw new AuthenticationException(
-                    "Invalid challenge value: expected "
-                            + (initialSalt + 1)
-                            + ", but got "
-                            + challenge);
         }
 
         @Override
         public boolean isCompleted() {
-            return isCompleted;
+            return status == Status.COMPLETED;
         }
     }
 
     private static class ServerAuthenticatorImpl implements ServerAuthenticator {
-        private boolean isCompleted = false;
-        private boolean firstAuthRequest = true;
-        private int initialSalt;
+        private enum Status {
+            RECEIVE_CLIENT_FIRST_MESSAGE,
+            RECEIVE_CLIENT_FINAL_MESSAGE,
+            COMPLETED
+        }
+
+        private Integer initialSalt;
+        private int retryNumber = 0;
+        private Status status = Status.RECEIVE_CLIENT_FIRST_MESSAGE;
 
         @Override
         public String protocol() {
             return MUTUAL_AUTH_PROTOCOL;
         }
 
+        @Override
+        public void initialize(AuthenticateContext context) {
+            this.status = Status.RECEIVE_CLIENT_FIRST_MESSAGE;
+            this.initialSalt = null;
+        }
+
         @Override
         public byte[] evaluateResponse(byte[] token) throws AuthenticationException {
             int tokenValue = parseToken(token);
-
-            // If mock error, return null.
-            if (isError(tokenValue, ErrorType.SERVER_NO_CHALLENGE)) {
-                return null;
-            }
-            if (isError(tokenValue, ErrorType.SERVER_ERROR_CHALLENGE)) {
-                return "-1".getBytes();
+            switch (status) {
+                case RECEIVE_CLIENT_FIRST_MESSAGE:
+                    if (isError(tokenValue, ErrorType.SERVER_NO_CHALLENGE)) {
+                        return null;
+                    }
+                    if (isError(tokenValue, ErrorType.SERVER_ERROR_CHALLENGE)) {
+                        return "-1".getBytes();
+                    }
+                    if (isError(tokenValue, ErrorType.RETRIABLE_EXCEPTION) && retryNumber++ < 3) {
+                        throw new RetriableAuthenticationException("Retriable exception");
+                    }
+
+                    initialSalt = tokenValue + 1;
+                    status = Status.RECEIVE_CLIENT_FINAL_MESSAGE;
+                    return String.valueOf(initialSalt).getBytes();
+                case RECEIVE_CLIENT_FINAL_MESSAGE:
+                    if (tokenValue == initialSalt + 1) {
+                        status = Status.COMPLETED;
+                        return new byte[0];
+                    } else {
+                        throw new IllegalArgumentException(
+                                "Invalid token value: expected "
+                                        + (initialSalt + 1)
+                                        + ", but got "
+                                        + tokenValue);
+                    }
+                default:
+                    return null;
             }
-
-            if (firstAuthRequest) {
-                initialSalt = tokenValue + 1;
-                firstAuthRequest = false;
-                return String.valueOf(initialSalt).getBytes();
-            }
-
-            if (tokenValue == initialSalt + 1) {
-                isCompleted = true;
-                return new byte[0];
-            }
-
-            throw new IllegalArgumentException(
-                    "Invalid token value: expected "
-                            + (initialSalt + 1)
-                            + ", but got "
-                            + tokenValue);
         }
 
         @Override
@@ -159,7 +197,7 @@ public FlussPrincipal createPrincipal() {
 
         @Override
         public boolean isCompleted() {
-            return isCompleted;
+            return status == Status.COMPLETED;
         }
     }
 
@@ -187,7 +225,8 @@ enum ErrorType {
         NONE(-1),
         SERVER_NO_CHALLENGE(-2),
         SERVER_ERROR_CHALLENGE(-3),
-        CLIENT_ERROR_SECOND_TOKEN(-4);
+        CLIENT_ERROR_SECOND_TOKEN(-4),
+        RETRIABLE_EXCEPTION(-5);
 
         final int code;
 


Reviews:
user: login: wuchong
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-10T09:10:33Z
user: login: loserwang1024
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T08:42:05Z
user: login: loserwang1024
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T08:43:02Z
user: login: loserwang1024
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T08:45:11Z
user: login: loserwang1024
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T08:45:35Z
user: login: loserwang1024
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T08:51:56Z
user: login: wuchong
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T03:05:16Z
user: login: wuchong
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-15T06:01:08Z

Review comments:
diff_hunk: @@ -319,7 +319,7 @@ private void handleAuthenticateRequest(
         if (!authenticator.isCompleted()) {
             byte[] token = authenticateRequest.getToken();
             byte[] challenge = authenticator.evaluateResponse(token);
-            if (!authenticator.isCompleted() && challenge != null) {
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/server/NettyServerHandler.java
user: login: wuchong
body: Is the removal of `!authenticator.isCompleted()` necessary? I though the `authenticator.evaluateResponse` may change status of `authenticator` into complete and don't need to send challenge. 
created_at: 2025-05-10T08:34:35Z
author_association: MEMBER
diff_hunk: @@ -366,63 +383,11 @@ private void handleApiVersionsResponse(ApiMessage response, Throwable cause) {
         synchronized (lock) {
             serverApiVersions =
                     new ServerApiVersions(((ApiVersionsResponse) response).getApiVersionsList());
-            LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
-            // send initial token
-            sendAuthenticate(new byte[0]);
-        }
-    }
-
-    private void sendAuthenticate(byte[] challenge) {
-        try {
-            if (!authenticator.isCompleted()) {
-                byte[] token = authenticator.authenticate(challenge);
-                if (token != null) {
-                    switchState(ConnectionState.AUTHENTICATING);
-                    AuthenticateRequest request =
-                            new AuthenticateRequest()
-                                    .setToken(token)
-                                    .setProtocol(authenticator.protocol());
-                    doSend(ApiKeys.AUTHENTICATE, request, new CompletableFuture<>(), true)
-                            .whenComplete(this::handleAuthenticateResponse);
-                    return;
-                }
-            }
-
-            assert authenticator.isCompleted();
-            switchState(ConnectionState.READY);
-
-        } catch (Exception e) {
-            LOG.error(
-                    "Authentication failed when authenticating challenge: {}",
-                    new String(challenge),
-                    e);
-            close(
-                    new FlussRuntimeException(
-                            "Authentication failed when authenticating challenge", e));
-        }
-    }
-
-    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
-        if (cause != null) {
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java
user: login: wuchong
body: Can we still authenticate in the `ServerConnection` instead of a separate channel handler? It increases the complexity that the `authenticator` and `switchState` out of the `lock` and may cause problems. 

I though we only need to re-auth here when the cause is retrable auth exception. In the future, we should also add re-auth logic here for expired authentications. We can also maintain backoff in `ServerConnection` as it is per-connection. 
created_at: 2025-05-10T08:49:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: wuchong
body: Is it necessary to remember the auth request? Can we just re-send the auth request from byte[0]? We should try to avoid maintaining states. 
created_at: 2025-05-10T08:54:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
+                    sendAuthenticateRequest(request);
+                    return;
+                }
+            }
+
+            assert authenticator.isCompleted();
+            callback.onAuthenticateSuccess();
+        } catch (Exception e) {
+            LOG.warn(
+                    "Authentication failed when authenticating challenge: {}",
+                    new String(challenge),
+                    e);
+            callback.onAuthenticateFailure(
+                    new FlussRuntimeException(
+                            "Authentication failed when authenticating challenge", e));
+        }
+    }
+
+    private void sendAuthenticateRequest(AuthenticateRequest request) {
+        callback.onAuthenticating(request).whenComplete(this::handleAuthenticateResponse);
+    }
+
+    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
+        if (cause != null) {
+            if (cause instanceof RetriableAuthenticationException
+                    && retryAuthNum < MAX_RETRY_AUTH_NUM) {
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: wuchong
body: We don't need to set max retry num and allow the request timeout by netty client. `Sender` also doesn't have max retry on `RetriableException`. 
created_at: 2025-05-10T08:58:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
+                    sendAuthenticateRequest(request);
+                    return;
+                }
+            }
+
+            assert authenticator.isCompleted();
+            callback.onAuthenticateSuccess();
+        } catch (Exception e) {
+            LOG.warn(
+                    "Authentication failed when authenticating challenge: {}",
+                    new String(challenge),
+                    e);
+            callback.onAuthenticateFailure(
+                    new FlussRuntimeException(
+                            "Authentication failed when authenticating challenge", e));
+        }
+    }
+
+    private void sendAuthenticateRequest(AuthenticateRequest request) {
+        callback.onAuthenticating(request).whenComplete(this::handleAuthenticateResponse);
+    }
+
+    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
+        if (cause != null) {
+            if (cause instanceof RetriableAuthenticationException
+                    && retryAuthNum < MAX_RETRY_AUTH_NUM) {
+                LOG.warn(
+                        "Authentication failed, retrying {} of {} times",
+                        retryAuthNum,
+                        MAX_RETRY_AUTH_NUM,
+                        cause);
+                retryAuthNum++;
+                context.executor()
+                        .schedule(
+                                () -> sendAuthenticateRequest(pendingRequest),
+                                calculateBackOff(),
+                                TimeUnit.MILLISECONDS);
+            } else {
+                LOG.warn("Authentication failed, and won't retry.", cause);
+                callback.onAuthenticateFailure(cause);
+            }
+            return;
+        }
+        if (!(response instanceof AuthenticateResponse)) {
+            callback.onAuthenticateFailure(
+                    new IllegalStateException("Unexpected response type " + response.getClass()));
+            return;
+        }
+
+        AuthenticateResponse authenticateResponse = (AuthenticateResponse) response;
+        if (authenticateResponse.hasChallenge()) {
+            generateTokenAndSendAuthenticate(((AuthenticateResponse) response).getChallenge());
+        } else if (authenticator.isCompleted()) {
+            callback.onAuthenticateSuccess();
+        } else {
+            callback.onAuthenticateFailure(
+                    new IllegalStateException(
+                            new IllegalStateException(
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: wuchong
body: Why nested `IllegalStateException`?
created_at: 2025-05-10T08:58:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: wuchong
body: `BASE_BACKOFF_MS` -> `100L` (better initial latency) and `MAX_BACKOFF_MS` -> `5s` (not idle too long)
created_at: 2025-05-10T09:09:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
+                    sendAuthenticateRequest(request);
+                    return;
+                }
+            }
+
+            assert authenticator.isCompleted();
+            callback.onAuthenticateSuccess();
+        } catch (Exception e) {
+            LOG.warn(
+                    "Authentication failed when authenticating challenge: {}",
+                    new String(challenge),
+                    e);
+            callback.onAuthenticateFailure(
+                    new FlussRuntimeException(
+                            "Authentication failed when authenticating challenge", e));
+        }
+    }
+
+    private void sendAuthenticateRequest(AuthenticateRequest request) {
+        callback.onAuthenticating(request).whenComplete(this::handleAuthenticateResponse);
+    }
+
+    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
+        if (cause != null) {
+            if (cause instanceof RetriableAuthenticationException
+                    && retryAuthNum < MAX_RETRY_AUTH_NUM) {
+                LOG.warn(
+                        "Authentication failed, retrying {} of {} times",
+                        retryAuthNum,
+                        MAX_RETRY_AUTH_NUM,
+                        cause);
+                retryAuthNum++;
+                context.executor()
+                        .schedule(
+                                () -> sendAuthenticateRequest(pendingRequest),
+                                calculateBackOff(),
+                                TimeUnit.MILLISECONDS);
+            } else {
+                LOG.warn("Authentication failed, and won't retry.", cause);
+                callback.onAuthenticateFailure(cause);
+            }
+            return;
+        }
+        if (!(response instanceof AuthenticateResponse)) {
+            callback.onAuthenticateFailure(
+                    new IllegalStateException("Unexpected response type " + response.getClass()));
+            return;
+        }
+
+        AuthenticateResponse authenticateResponse = (AuthenticateResponse) response;
+        if (authenticateResponse.hasChallenge()) {
+            generateTokenAndSendAuthenticate(((AuthenticateResponse) response).getChallenge());
+        } else if (authenticator.isCompleted()) {
+            callback.onAuthenticateSuccess();
+        } else {
+            callback.onAuthenticateFailure(
+                    new IllegalStateException(
+                            new IllegalStateException(
+                                    "client authenticator is not completed while server generate no challenge.")));
+        }
+    }
+
+    private long calculateBackOff() {
+        // Exponential backoff with jitter
+        long backOff = BASE_BACKOFF_MS * (1L << retryAuthNum);
+        backOff = Math.min(backOff, MAX_BACKOFF_MS); // Ensure it does not exceed maxBackOffMs
+        long jitter = (long) (backOff * JITTER_FACTOR * RANDOM.nextDouble());
+        return backOff + jitter;
+    }
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: wuchong
body: Introduce Kafka `ExponentialBackoff` and `ExponentialBackoffTest` for the verified backoff util. The jitter should consider -20% ~ + 20%.
created_at: 2025-05-10T09:10:21Z
author_association: MEMBER
diff_hunk: @@ -319,7 +319,7 @@ private void handleAuthenticateRequest(
         if (!authenticator.isCompleted()) {
             byte[] token = authenticateRequest.getToken();
             byte[] challenge = authenticator.evaluateResponse(token);
-            if (!authenticator.isCompleted() && challenge != null) {
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/server/NettyServerHandler.java
user: login: loserwang1024
body: For some authentication protocol, the server will be complete before the client becomes complete.

Take SCRAM-SHA-256  of kafka  for example,
1. sasl server recieves RECEIVE_CLIENT_FINAL_MESSAGE, and then mark authenticator server status as COMPLETE, then still send serverFinalMessage 
```java
// org.apache.kafka.common.security.scram.internals.ScramSaslServer#evaluateResponse
case RECEIVE_CLIENT_FINAL_MESSAGE:
                    try {
                        ClientFinalMessage clientFinalMessage = new ClientFinalMessage(response);
                        verifyClientProof(clientFinalMessage);
                        byte[] serverKey = scramCredential.serverKey();
                        byte[] serverSignature = formatter.serverSignature(serverKey, clientFirstMessage, serverFirstMessage, clientFinalMessage);
                        ServerFinalMessage serverFinalMessage = new ServerFinalMessage(null, serverSignature);
                        clearCredentials();
                        setState(State.COMPLETE);
                        return serverFinalMessage.toBytes();
                    } catch (InvalidKeyException e) {
                        throw new SaslException("Authentication failed: Invalid client final message", e);
                    }
```
2. Then sasl client recieve this RECEIVE_SERVER_FINAL_MESSAGE, then will set status and State.COMPLETE and then return null(no longer send anymore)
```java
org.apache.kafka.common.security.scram.internals.ScramSaslClient#evaluateChallenge

 case RECEIVE_SERVER_FINAL_MESSAGE:
                    ServerFinalMessage serverFinalMessage = new ServerFinalMessage(challenge);
                    if (serverFinalMessage.error() != null)
                        throw new SaslException("Sasl authentication using " + mechanism + " failed with error: " + serverFinalMessage.error());
                    handleServerFinalMessage(serverFinalMessage.serverSignature());
                    setState(State.COMPLETE);
                    return null;
```

I add same logic in com.alibaba.fluss.rpc.netty.authenticate.MutualAuthenticationPlugin.

created_at: 2025-05-12T08:42:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -366,63 +383,11 @@ private void handleApiVersionsResponse(ApiMessage response, Throwable cause) {
         synchronized (lock) {
             serverApiVersions =
                     new ServerApiVersions(((ApiVersionsResponse) response).getApiVersionsList());
-            LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
-            // send initial token
-            sendAuthenticate(new byte[0]);
-        }
-    }
-
-    private void sendAuthenticate(byte[] challenge) {
-        try {
-            if (!authenticator.isCompleted()) {
-                byte[] token = authenticator.authenticate(challenge);
-                if (token != null) {
-                    switchState(ConnectionState.AUTHENTICATING);
-                    AuthenticateRequest request =
-                            new AuthenticateRequest()
-                                    .setToken(token)
-                                    .setProtocol(authenticator.protocol());
-                    doSend(ApiKeys.AUTHENTICATE, request, new CompletableFuture<>(), true)
-                            .whenComplete(this::handleAuthenticateResponse);
-                    return;
-                }
-            }
-
-            assert authenticator.isCompleted();
-            switchState(ConnectionState.READY);
-
-        } catch (Exception e) {
-            LOG.error(
-                    "Authentication failed when authenticating challenge: {}",
-                    new String(challenge),
-                    e);
-            close(
-                    new FlussRuntimeException(
-                            "Authentication failed when authenticating challenge", e));
-        }
-    }
-
-    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
-        if (cause != null) {
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java
user: login: loserwang1024
body: Ok, Done it. Because this modification is different with last commit, thus rebase with a new commit.
created_at: 2025-05-12T08:43:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: loserwang1024
body: ok, why resending the request  is because that authenticator is with status. If send byte[0], status will be error. 

I add a com.alibaba.fluss.security.auth.ServerAuthenticator#initialize and com.alibaba.fluss.security.auth.ClientAuthenticator#initialize to initialize and reinitialize the authenticator. When retriable exception occurs, will reinitialize the authenticator and send initial byte[0].
created_at: 2025-05-12T08:45:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
+                    sendAuthenticateRequest(request);
+                    return;
+                }
+            }
+
+            assert authenticator.isCompleted();
+            callback.onAuthenticateSuccess();
+        } catch (Exception e) {
+            LOG.warn(
+                    "Authentication failed when authenticating challenge: {}",
+                    new String(challenge),
+                    e);
+            callback.onAuthenticateFailure(
+                    new FlussRuntimeException(
+                            "Authentication failed when authenticating challenge", e));
+        }
+    }
+
+    private void sendAuthenticateRequest(AuthenticateRequest request) {
+        callback.onAuthenticating(request).whenComplete(this::handleAuthenticateResponse);
+    }
+
+    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
+        if (cause != null) {
+            if (cause instanceof RetriableAuthenticationException
+                    && retryAuthNum < MAX_RETRY_AUTH_NUM) {
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: loserwang1024
body: remove it
created_at: 2025-05-12T08:45:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.rpc.netty.client;
+
+import com.alibaba.fluss.exception.FlussRuntimeException;
+import com.alibaba.fluss.exception.RetriableAuthenticationException;
+import com.alibaba.fluss.rpc.messages.ApiMessage;
+import com.alibaba.fluss.rpc.messages.AuthenticateRequest;
+import com.alibaba.fluss.rpc.messages.AuthenticateResponse;
+import com.alibaba.fluss.security.auth.ClientAuthenticator;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelHandlerContext;
+import com.alibaba.fluss.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Implementation of the channel handler to send authenticator. The client handler is not shared,
+ * there is a client handler instance for each channel connection.
+ */
+public class NettyAuthenticationHandler extends ChannelInboundHandlerAdapter {
+    private static final Logger LOG = LoggerFactory.getLogger(NettyAuthenticationHandler.class);
+    private static final Random RANDOM = new Random();
+    private static final long BASE_BACKOFF_MS = 500L;
+    private static final long MAX_BACKOFF_MS = 60000L;
+    // Jitter factor for randomization
+    private static final double JITTER_FACTOR = 0.2;
+    private static final int MAX_RETRY_AUTH_NUM = 10;
+
+    private final AuthenticatorHandlerCallback callback;
+    private final ClientAuthenticator authenticator;
+
+    private int retryAuthNum = 0;
+    private ChannelHandlerContext context;
+    private AuthenticateRequest pendingRequest;
+
+    public NettyAuthenticationHandler(
+            AuthenticatorHandlerCallback callback, ClientAuthenticator authenticator) {
+        this.callback = callback;
+        this.authenticator = authenticator;
+    }
+
+    @Override
+    public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
+        // send initial authentication token.
+        context = ctx;
+        LOG.debug("Begin to authenticate with protocol {}", authenticator.protocol());
+        generateTokenAndSendAuthenticate(new byte[0]);
+    }
+
+    private void generateTokenAndSendAuthenticate(byte[] challenge) {
+        try {
+            if (!authenticator.isCompleted()) {
+                byte[] token = authenticator.authenticate(challenge);
+                if (token != null) {
+                    AuthenticateRequest request =
+                            new AuthenticateRequest()
+                                    .setToken(token)
+                                    .setProtocol(authenticator.protocol());
+                    pendingRequest = request;
+                    sendAuthenticateRequest(request);
+                    return;
+                }
+            }
+
+            assert authenticator.isCompleted();
+            callback.onAuthenticateSuccess();
+        } catch (Exception e) {
+            LOG.warn(
+                    "Authentication failed when authenticating challenge: {}",
+                    new String(challenge),
+                    e);
+            callback.onAuthenticateFailure(
+                    new FlussRuntimeException(
+                            "Authentication failed when authenticating challenge", e));
+        }
+    }
+
+    private void sendAuthenticateRequest(AuthenticateRequest request) {
+        callback.onAuthenticating(request).whenComplete(this::handleAuthenticateResponse);
+    }
+
+    private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
+        if (cause != null) {
+            if (cause instanceof RetriableAuthenticationException
+                    && retryAuthNum < MAX_RETRY_AUTH_NUM) {
+                LOG.warn(
+                        "Authentication failed, retrying {} of {} times",
+                        retryAuthNum,
+                        MAX_RETRY_AUTH_NUM,
+                        cause);
+                retryAuthNum++;
+                context.executor()
+                        .schedule(
+                                () -> sendAuthenticateRequest(pendingRequest),
+                                calculateBackOff(),
+                                TimeUnit.MILLISECONDS);
+            } else {
+                LOG.warn("Authentication failed, and won't retry.", cause);
+                callback.onAuthenticateFailure(cause);
+            }
+            return;
+        }
+        if (!(response instanceof AuthenticateResponse)) {
+            callback.onAuthenticateFailure(
+                    new IllegalStateException("Unexpected response type " + response.getClass()));
+            return;
+        }
+
+        AuthenticateResponse authenticateResponse = (AuthenticateResponse) response;
+        if (authenticateResponse.hasChallenge()) {
+            generateTokenAndSendAuthenticate(((AuthenticateResponse) response).getChallenge());
+        } else if (authenticator.isCompleted()) {
+            callback.onAuthenticateSuccess();
+        } else {
+            callback.onAuthenticateFailure(
+                    new IllegalStateException(
+                            new IllegalStateException(
+                                    "client authenticator is not completed while server generate no challenge.")));
+        }
+    }
+
+    private long calculateBackOff() {
+        // Exponential backoff with jitter
+        long backOff = BASE_BACKOFF_MS * (1L << retryAuthNum);
+        backOff = Math.min(backOff, MAX_BACKOFF_MS); // Ensure it does not exceed maxBackOffMs
+        long jitter = (long) (backOff * JITTER_FACTOR * RANDOM.nextDouble());
+        return backOff + jitter;
+    }
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/NettyAuthenticationHandler.java
user: login: loserwang1024
body: done it.
created_at: 2025-05-12T08:51:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,4 +83,9 @@ public interface ServerAuthenticator {
      * complete).
      */
     FlussPrincipal createPrincipal();
+
+    /** The context of the authentication process. */
+    interface AuthenticateContext {
+        Channel channel();
path: fluss-common/src/main/java/com/alibaba/fluss/security/auth/ServerAuthenticator.java
user: login: wuchong
body: remove it as it is never used. 
created_at: 2025-05-13T03:02:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,91 @@
+/*
+ * Copyright (c) 2025 Alibaba Group Holding Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.alibaba.fluss.utils;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.util.concurrent.ThreadLocalRandom;
+
+/* This file is based on source code of Apache Kafka Project (https://kafka.apache.org/), licensed by the Apache
+ * Software Foundation (ASF) under the Apache License, Version 2.0. See the NOTICE file distributed with this work for
+ * additional information regarding copyright ownership. */
+
+/**
+ * A utility class for computing exponential backoff values, commonly used for retry logic,
+ * reconnection attempts, and timeout management.
+ *
+ * <p>The backoff interval increases exponentially with each attempt, following the formula:
+ *
+ * <pre>
+ * Backoff(attempts) = random(1 - jitter, 1 + jitter) * initialInterval * multiplier^attempts
+ * </pre>
+ *
+ * <p>If {@code maxInterval} is less than {@code initialInterval}, a constant backoff of {@code
+ * maxInterval} will be applied. The jitter factor ensures randomness to avoid thundering herd
+ * problems, but will never cause the result to exceed the configured maximum interval.
+ *
+ * <p>Instances of this class are thread-safe and can be shared across multiple threads.
+ */
+@ThreadSafe
+public class ExponentialBackoff {
path: fluss-common/src/main/java/com/alibaba/fluss/utils/ExponentialBackoff.java
user: login: wuchong
body: Add the file path `./fluss-common/src/main/java/com/alibaba/fluss/utils/ExponentialBackoff.java` to LICENSE in root. 
created_at: 2025-05-13T03:04:12Z
author_association: MEMBER
diff_hunk: @@ -404,7 +417,16 @@ private void sendAuthenticate(byte[] challenge) {
 
     private void handleAuthenticateResponse(ApiMessage response, Throwable cause) {
         if (cause != null) {
-            close(cause);
+            if (cause instanceof RetriableAuthenticationException) {
+                LOG.warn("Authentication failed, retrying {} times", retryAuthCount, cause);
+                channel.eventLoop()
+                        .schedule(
+                                this::sendInitialToken,
+                                backoff.backoff(retryAuthCount++),
path: fluss-rpc/src/main/java/com/alibaba/fluss/rpc/netty/client/ServerConnection.java
user: login: wuchong
body: We should add a big lock synchronized for the whole `handleAuthenticateResponse` method. 
created_at: 2025-05-13T03:04:43Z
author_association: MEMBER

PR comments:
user: login: loserwang1024
created_at: 2025-05-12T08:55:48Z
body: Rebase this pr with three modification:
1. Add ExponentialBackoff to calculate the backoff time.
2. Retain authentication logic in ServerConnection.
3. Add `initialize ` method for ServerAuthenticator and ClientAuthenticator to reset the status of authenticator.
author_association: CONTRIBUTOR
user: login: luoyuxia
created_at: 2025-05-13T03:14:56Z
body: Maybe we can verify this pr in our env before merge that to avoid go back and forth. Maybe in this afternoon or tomorrow to verify this pr.
author_association: CONTRIBUTOR
user: login: loserwang1024
created_at: 2025-05-13T03:36:20Z
body: > Maybe we can verify this pr in our env before merge that to avoid go back and forth. Maybe in this afternoon or tomorrow to verify this pr.

ok, I will apply the modification of this pr to env test.
author_association: CONTRIBUTOR
user: login: wuchong
created_at: 2025-05-15T06:01:34Z
body: @loserwang1024 please ping me when you have verified in internal env. 
author_association: MEMBER
user: login: wuchong
created_at: 2025-05-19T02:37:35Z
body: It is passed in our internal env. 
author_association: MEMBER