Repository: provectus/kafka-ui
Issue: Cluster order is not respected
Labels: type/bug
scope/backend
status/accepted
status/confirmed

**Describe the bug**

Previously we had only 3 clusters defined in Docker service config, with that the order was preserved (from 0 to 2 as per env vars index number). After evaluation we have added remaining 12 clusters (yes, we have total of 15 Kafka clusters). The order now is completely random. **This happens in both nav menu (left side of UI with vertical list) and in dashboard view**.

**Set up**
Docker off of main branch using `master` tag in Docker hub.


**Steps to Reproduce**
Steps to reproduce the behavior:

1. Add 15 or more clusters to config using user-friendly name in `KAFKA_CLUSTERS_0_NAME:`
2. Look at UI

**Expected behavior**
Order is preserved as in Config

**Actual behavior**
Order seems to be random

**Screenshot**
Cannot share because of private information.

**Additional context**
It is definitely not ordered by anything else. I checked, it does not follow index ordering, nor alphabetic ordering, it is just random, but once the "random" order is shown this exact ordering will persist between reloads.


Issue comments:
@akamensky fixed, please try master version
@iliax I've tried the latest master version, but I get this error on service start (and it crashes):

```
05:05:36.501 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clustersMetricsScheduler': Invocation of init method failed; nested exception is org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1638075933368, tries=1, nextAllowedTryMs=1638075933469) timed out at 1638075933369 after 1 attempt(s)
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.12.jar!/:5.3.12]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.12.jar!/:5.3.12]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.12.jar!/:5.3.12]
	at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:64) ~[spring-boot-2.5.6.jar!/:2.5.6]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) [spring-boot-2.5.6.jar!/:2.5.6]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) [spring-boot-2.5.6.jar!/:2.5.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) [spring-boot-2.5.6.jar!/:2.5.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) [spring-boot-2.5.6.jar!/:2.5.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) [spring-boot-2.5.6.jar!/:2.5.6]
	at com.provectus.kafka.ui.KafkaUiApplication.main(KafkaUiApplication.java:14) [classes!/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [kafka-ui-api.jar:?]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [kafka-ui-api.jar:?]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [kafka-ui-api.jar:?]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [kafka-ui-api.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1638075933368, tries=1, nextAllowedTryMs=1638075933469) timed out at 1638075933369 after 1 attempt(s)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.4.11.jar!/:3.4.11]
		at reactor.core.publisher.Mono.block(Mono.java:1706) ~[reactor-core-3.4.11.jar!/:3.4.11]
		at com.provectus.kafka.ui.service.ClustersMetricsScheduler.updateMetrics(ClustersMetricsScheduler.java:35) ~[classes!/:?]
		at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
		at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
		at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
		at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
		at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.12.jar!/:5.3.12]
		at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.12.jar!/:5.3.12]
		at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.12.jar!/:5.3.12]
		at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:64) ~[spring-boot-2.5.6.jar!/:2.5.6]
		at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) [spring-boot-2.5.6.jar!/:2.5.6]
		at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) [spring-boot-2.5.6.jar!/:2.5.6]
		at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) [spring-boot-2.5.6.jar!/:2.5.6]
		at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) [spring-boot-2.5.6.jar!/:2.5.6]
		at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) [spring-boot-2.5.6.jar!/:2.5.6]
		at com.provectus.kafka.ui.KafkaUiApplication.main(KafkaUiApplication.java:14) [classes!/:?]
		at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
		at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
		at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
		at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
		at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [kafka-ui-api.jar:?]
		at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [kafka-ui-api.jar:?]
		at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [kafka-ui-api.jar:?]
		at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [kafka-ui-api.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
```
@akamensky 
In current master we do first metrics collection on application startup and fail if we cant handle any of clusters - I [just fixed ](https://github.com/provectus/kafka-ui/pull/1136) logging issue - please refresh master and try again - error message should be more descriptive. 

BTW: this eager initialization can be a problem in some situations when you have a lot of cluster, we will discuss and maybe add option to disable it. 
@iliax I think at least it shouldn't crash if any of the clusters are unreachable. Logging that is good. Showing in UI a red dot instead of green (as it was working previously) even better. But definitely shouldn't crash with exception IMO. 
@akamensky thank you for feedback, we will rework initialization soon
Thanks, looks like with the fix it works as expected now -- the error messages shown in log, and the application is accessible via UI.

As for the original issue -- ordering looks good now. Thanks @iliax 

Pull request review comments:

Patch that resolved the issue:
diff --git a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersMetricsScheduler.java b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersMetricsScheduler.java
index 0555243bb17..712077d604b 100644
--- a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersMetricsScheduler.java
+++ b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersMetricsScheduler.java
@@ -1,6 +1,5 @@
 package com.provectus.kafka.ui.service;
 
-import java.util.Map;
 import javax.annotation.PostConstruct;
 import lombok.RequiredArgsConstructor;
 import lombok.extern.log4j.Log4j2;
@@ -24,10 +23,9 @@ public class ClustersMetricsScheduler {
       initialDelayString = "${kafka.update-metrics-rate-millis:30000}"
   )
   public void updateMetrics() {
-    Flux.fromIterable(clustersStorage.getKafkaClustersMap().entrySet())
+    Flux.fromIterable(clustersStorage.getKafkaClusters())
         .parallel()
         .runOn(Schedulers.parallel())
-        .map(Map.Entry::getValue)
         .flatMap(cluster -> {
           log.debug("Start getting metrics for kafkaCluster: {}", cluster.getName());
           return metricsService.updateCache(cluster)
diff --git a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersStorage.java b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersStorage.java
index b77135f034b..08b71b4d318 100644
--- a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersStorage.java
+++ b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ClustersStorage.java
@@ -1,33 +1,22 @@
 package com.provectus.kafka.ui.service;
 
+import com.google.common.collect.ImmutableMap;
 import com.provectus.kafka.ui.config.ClustersProperties;
 import com.provectus.kafka.ui.mapper.ClusterMapper;
 import com.provectus.kafka.ui.model.KafkaCluster;
 import java.util.Collection;
-import java.util.Map;
 import java.util.Optional;
-import java.util.concurrent.ConcurrentHashMap;
-import javax.annotation.PostConstruct;
-import lombok.RequiredArgsConstructor;
-import org.mapstruct.factory.Mappers;
 import org.springframework.stereotype.Component;
 
 @Component
-@RequiredArgsConstructor
 public class ClustersStorage {
 
-  private final Map<String, KafkaCluster> kafkaClusters = new ConcurrentHashMap<>();
+  private final ImmutableMap<String, KafkaCluster> kafkaClusters;
 
-  private final ClustersProperties clusterProperties;
-
-  private final ClusterMapper clusterMapper = Mappers.getMapper(ClusterMapper.class);
-
-  @PostConstruct
-  public void init() {
-    for (ClustersProperties.Cluster clusterProperties : clusterProperties.getClusters()) {
-      KafkaCluster cluster = clusterMapper.toKafkaCluster(clusterProperties);
-      kafkaClusters.put(clusterProperties.getName(), cluster);
-    }
+  public ClustersStorage(ClustersProperties properties, ClusterMapper mapper) {
+    var builder = ImmutableMap.<String, KafkaCluster>builder();
+    properties.getClusters().forEach(c -> builder.put(c.getName(), mapper.toKafkaCluster(c)));
+    this.kafkaClusters = builder.build();
   }
 
   public Collection<KafkaCluster> getKafkaClusters() {
@@ -37,8 +26,4 @@ public Collection<KafkaCluster> getKafkaClusters() {
   public Optional<KafkaCluster> getClusterByName(String clusterName) {
     return Optional.ofNullable(kafkaClusters.get(clusterName));
   }
-
-  public Map<String, KafkaCluster> getKafkaClustersMap() {
-    return kafkaClusters;
-  }
 }


Test patch: