Repository: provectus/kafka-ui
Issue: Timeouts are not configurable in envvars
Labels: type/bug
status/pending
scope/backend
status/accepted

**Describe the bug**

Default timeouts are way too short and there is no documented way to configure them when running from official Docker image. In our system only clusters that are physically nearby the UI are "online", the rest are all "offline" with logs:

```
01:35:06.481 [kafka-admin-client-thread | adminclient-11] ERROR com.provectus.kafka.ui.service.MetricsService - Failed to collect cluster CLUSTER_NAME info
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1638927306473, tries=12, nextAllowedTryMs=1638927306574) timed out at 1638927306474 after 12 attempt(s)
```

Meantime I can use Kafka own cli tools just fine to connect to those and get all information that I need, so it is just that timeouts are way too short and there is not way to configure them.


**Set up**

* Globally distributed large number of clusters
* Single UI to manage all of them
* Private (MPLS) networking between sites


**Steps to Reproduce**
Steps to reproduce the behavior:

1. Have a cluster that physically on the other side of the planet
2. Run cluster in Docker swarm using official "master" image
2. Add clusters to UI cluster list using env vars
3. ....
4. Observe timeouts on every connect

**Expected behavior**
Works

**Actual behavior**
Timeouts on remote clusters

**Screenshots**
N/A


**Additional context**
N/A


Issue comments:
Looking at YAML configuration file I've tried to use following in Docker stack envvars:

```
      KAFKA_ADMIN-CLIENT-TIMEOUT: 30000
      ZOOKEEPER_CONNECTION-TIMEOUT: 30000
```

Which works, but again for some clusters it timeouts (but fewer this time). I don't understand why it needs such a high value, because in fact the connections are not that slow to those clusters, bundled-in Kafka scripts complete their requests in under a second to the most remote cluster (tested from the same host where UI runs).

It looks to me something with connection logic is off here.
@akamensky please try to incerease `KAFKA_ADMIN-CLIENT-TIMEOUT` to 60000 or more and check behaviour. Note that it is not connection timeout - it is API operation timeout.  If it will help it mean that problem is in high latency. Otherwise maybe some connections error can take place (like closed ports or smth). 
@iliax yes, already done that and it does work with 60000, but it is still quite high value I feel and perhaps default value is too low.

But overall I think this issue is more about documenting the configuration options and perhaps make them uniform (replace `-` with `_` for envvar options). But would be nice to perhaps raise default values as well to something higher as some operations may be slow. In our case it is the network link speed, because this connection goes over "management" link, which is slow (to leave capacity for actual data link).
@akamensky I increased default timeout to 30s (it was 5s) and added doc for this property

Pull request review comments:

Patch that resolved the issue:
diff --git a/README.md b/README.md
index dcfa9f652a5..7eb9a4f5859 100644
--- a/README.md
+++ b/README.md
@@ -162,7 +162,11 @@ For example, if you want to use an environment variable to set the `name` parame
 
 |Name               	|Description
 |-----------------------|-------------------------------
-|`SERVER_SERVLET_CONTEXT_PATH` | URI basePath
+|`SERVER_SERVLET_CONTEXT_PATH`  | URI basePath
+|`LOGGING_LEVEL_ROOT`        	| Setting log level (trace, debug, info, warn, error, fatal, off). Default: debug
+|`LOGGING_LEVEL_COM_PROVECTUS`  |Setting log level (trace, debug, info, warn, error, fatal, off). Default: debug
+|`SERVER_PORT` |Port for the embedded server. Default: `8080`
+|`KAFKA_ADMIN-CLIENT-TIMEOUT` | Kafka API timeout in ms. Default: `30000`
 |`KAFKA_CLUSTERS_0_NAME` | Cluster name
 |`KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS` 	|Address where to connect 
 |`KAFKA_CLUSTERS_0_ZOOKEEPER` 	| Zookeeper service address 
@@ -177,9 +181,6 @@ For example, if you want to use an environment variable to set the `name` parame
 |`KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION`        	|Disable collecting segments information. It should be true for confluent cloud. Default: false
 |`KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME` |Given name for the Kafka Connect cluster
 |`KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS` |Address of the Kafka Connect service endpoint 
-|`LOGGING_LEVEL_ROOT`        	| Setting log level (trace, debug, info, warn, error, fatal, off). Default: debug
-|`LOGGING_LEVEL_COM_PROVECTUS`        	|Setting log level (trace, debug, info, warn, error, fatal, off). Default: debug
-|`SERVER_PORT` |Port for the embedded server. Default `8080`
 |`KAFKA_CLUSTERS_0_JMXSSL` |Enable SSL for JMX? `true` or `false`. For advanced setup, see `kafka-ui-jmx-secured.yml`
 |`KAFKA_CLUSTERS_0_JMXUSERNAME` |Username for JMX authentication
 |`KAFKA_CLUSTERS_0_JMXPASSWORD` |Password for JMX authentication
diff --git a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/AdminClientServiceImpl.java b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/AdminClientServiceImpl.java
index 88cc259b999..f92ea1e1d5b 100644
--- a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/AdminClientServiceImpl.java
+++ b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/AdminClientServiceImpl.java
@@ -20,7 +20,7 @@
 public class AdminClientServiceImpl implements AdminClientService, Closeable {
   private final Map<String, ReactiveAdminClient> adminClientCache = new ConcurrentHashMap<>();
   @Setter // used in tests
-  @Value("${kafka.admin-client-timeout}")
+  @Value("${kafka.admin-client-timeout:30000}")
   private int clientTimeout;
 
   @Override
diff --git a/kafka-ui-api/src/main/resources/application-local.yml b/kafka-ui-api/src/main/resources/application-local.yml
index 93a5745c388..54bdb59f5b5 100644
--- a/kafka-ui-api/src/main/resources/application-local.yml
+++ b/kafka-ui-api/src/main/resources/application-local.yml
@@ -27,9 +27,6 @@ kafka:
   #      protobufMessageNameByTopic:
   #        input-topic: InputMessage
   #        output-topic: OutputMessage
-  admin-client-timeout: 5000
-zookeeper:
-  connection-timeout: 1000
 spring:
   jmx:
     enabled: true
diff --git a/kafka-ui-api/src/main/resources/application-sdp.yml b/kafka-ui-api/src/main/resources/application-sdp.yml
index 2de30ccb7ca..f8d4445fedb 100644
--- a/kafka-ui-api/src/main/resources/application-sdp.yml
+++ b/kafka-ui-api/src/main/resources/application-sdp.yml
@@ -11,6 +11,3 @@ kafka:
   #      zookeeper: zookeeper1:2181
   #      bootstrapServers: kafka1:29092
   #      schemaRegistry: http://schemaregistry1:8085
-  admin-client-timeout: 5000
-zookeeper:
-  connection-timeout: 1000
diff --git a/kafka-ui-api/src/main/resources/application.yml b/kafka-ui-api/src/main/resources/application.yml
index 2cda55ac2a9..54d4c090117 100644
--- a/kafka-ui-api/src/main/resources/application.yml
+++ b/kafka-ui-api/src/main/resources/application.yml
@@ -1,7 +1,3 @@
-kafka:
-  admin-client-timeout: 5000
-zookeeper:
-  connection-timeout: 1000
 auth:
   enabled: false
 management:
@@ -13,4 +9,4 @@ management:
   endpoints:
     web:
       exposure:
-        include: "info,health"
+        include: "info,health"
\ No newline at end of file


Test patch:
diff --git a/kafka-ui-api/src/test/resources/application-test.yml b/kafka-ui-api/src/test/resources/application-test.yml
index ec67d666a01..63dada67238 100644
--- a/kafka-ui-api/src/test/resources/application-test.yml
+++ b/kafka-ui-api/src/test/resources/application-test.yml
@@ -6,9 +6,6 @@ kafka:
       zookeeper: localhost:2181
       schemaRegistry: http://localhost:8081
       jmxPort: 9997
-  admin-client-timeout: 5000
-zookeeper:
-  connection-timeout: 1000
 spring:
   jmx:
     enabled: true