Repository: provectus/kafka-ui
Issue: Can't connect to azure event hub kafka interface 
Labels: type/bug
scope/backend
status/accepted
status/confirmed

<!--

Don't forget to check for existing issues/discussions regarding your proposal. We might already have it.
https://github.com/provectus/kafka-ui/issues
https://github.com/provectus/kafka-ui/discussions

-->

**Describe the bug**
<!--(A clear and concise description of what the bug is.)-->
We wanted to monitor our azure event hub, which we mainly use as a kafka server.
The kafka interface works and our backend services can communicate with the kafka interface

Monitoring also works with kafka-tool/offset-explorer

But kafka-ui fails to connect to the hub with this error message:
```
2022-08-12 14:27:25,701 ERROR [parallel-6] c.p.k.u.s.MetricsService: Failed to collect cluster prod-event-hub info
java.lang.IllegalStateException: Error while creating AdminClient for Cluster prod-event-hub
        at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$3(AdminClientServiceImpl.java:45)
        at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3733)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)
        at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)
        at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:187)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
```

The kafka UI has full admin rights on the assigned user

**Set up**
<!--
How do you run the app? Please provide as much info as possible:
1. App version (docker image version or check commit hash in the top left corner in UI)
2. Helm chart version, if you use one
3. Any IAAC configs

We might close the issue without further explanation if you don't provide such information.
-->
Kafka-UI is setup with a standard docker compose:
```yaml
version: '2.7'

services:
  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8085:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=prod-event-hub
      - KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=***.servicebus.windows.net:9093
      - KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN
      - KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username='$$ConnectionString' password='***;
```

**Expected behavior**
<!--
(A clear and concise description of what you expected to happen)
-->
A functioning connection with the azure event hub



Issue comments:
Hello there WizzardMaker! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀
Hey, thanks for reaching out. We'll take a look
Hi,

Im interested on the connection to Azure Events Hub too. 

Thanks!!
Seems like they [don't care ](https://github.com/Azure/azure-event-hubs-for-kafka/issues/61) that much about implementing describeConfigs.

TODO: look for a workaround
Thanks for the feedback @Haarolean !
@Haarolean I would like to ask current kafka ui version whether have supported to connect Azure Eventhub? Thank you.
@hueiyuan thanks for the reminder

Might be doable now considering #3376,

@iliax PTAL
@Haarolean Thanks for your reply. so maybe can I connect eventhub in `release 0.5.0`?
By the way, also please @iliax to confirm it. thank you.
@hueiyuan no you can't since this issue is still open.
The same problem here with MSK(AWS) Kafka 1.1.1.
@robsonpeixoto MSK (even serverless) is fully supported. Try `master`-labeled docker image and if it doesn't work, raise a new issue/discussion.
@robsonpeixoto https://aws.amazon.com/marketplace/pp/prodview-ogtt5hfhzkq6a
I'm having the same issues, and using the `master` tagged Docker image doesn't help.
The same `UnknownTopicOrPartitionException: This server does not host this topic-partition` error keeps popping up unfortunately.
@FireDrunk fortunately, we're already working on this :)
@WizzardMaker @FireDrunk @hueiyuan hey guys, can you please verify it works for you on `master` - tagged docker image?
Can confirm that it works with the newest master branch image
@iliax @WizzardMaker 
After verifying, my kafka ui still does not connect eventhub. Could you give connect example of docker-compose.yaml ? My error message is below:
```
kafka-ui  | 2023-03-27 01:02:12,188 WARN  [kafka-admin-client-thread | kafka-ui-admin-1679878922-2] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1679878922-2] Connection to node -1 (<my-eventhub-namespace>.servicebus.windows.net/40.78.253.136:9093) could not be established. Broker may not be available.
```

I also provide my docker-compose.yaml for your reference:
```yaml
version: '3'

services:
  kafka-ui:
    image: provectuslabs/kafka-ui:master
    container_name: kafka-ui
    ports:
      - "8080:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=prod-event-hub
      - KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=<my-eventhub-namespaces>.servicebus.windows.net:9093
      - KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN
      - KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username='$ConnectionString' password='Endpoint=sb://<my-eventhub-namespaces>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<my-key>';
```
> @iliax @WizzardMaker After verifying, my kafka ui still does not connect eventhub. Could you give connect example of docker-compose.yaml ? My error message is below:
> 
> ```
> kafka-ui  | 2023-03-27 01:02:12,188 WARN  [kafka-admin-client-thread | kafka-ui-admin-1679878922-2] o.a.k.c.NetworkClient: [AdminClient clientId=kafka-ui-admin-1679878922-2] Connection to node -1 (<my-eventhub-namespace>.servicebus.windows.net/40.78.253.136:9093) could not be established. Broker may not be available.
> ```
> 
> I also provide my docker-compose.yaml for your reference:
> 
> ```yaml
> version: '3'
> 
> services:
>   kafka-ui:
>     image: provectuslabs/kafka-ui:master
>     container_name: kafka-ui
>     ports:
>       - "8080:8080"
>     restart: always
>     environment:
>       - KAFKA_CLUSTERS_0_NAME=prod-event-hub
>       - KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
>       - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=<my-eventhub-namespaces>.servicebus.windows.net:9093
>       - KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN
>       - KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username='$ConnectionString' password='Endpoint=sb://<my-eventhub-namespaces>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<my-key>';
> ```

That error message looks more like a general connection problem, and not the problem discussed in this issue.

Are you sure, that your at all able to connect to your event hub at that address? (What is \<my-eventhub-namespaces\>? Make sure that this is the name of the Azure resource - Check your "Host name" Property of the Event Hub Resource)
@WizzardMaker 
`<my-eventhub-namespaces>` just is a namespace of eventhub. You can image this is a variable which can be replaced. Because I do not want to paste complete host name here. Based on this reason, I have confirm my host name of eventhub is correct. So I not sure and want to confirm this configure of docker-compose.yaml whether have incorrect setting.
The configuration looks correct.

Is the network of your docker container restricted in any way? Can you open a shell on the container and ping to the internet (`ping google.com` for example)?
@hueiyuan also make sure that you are using standart+ plan for Eventhub, since lower versions are not kafka-protocol compatible
Worked like a charm for me. Great job, Thanks!

Pull request review comments:

Patch that resolved the issue:
diff --git a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ReactiveAdminClient.java b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ReactiveAdminClient.java
index 19d06a0c48b..7cabb79f2d0 100644
--- a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ReactiveAdminClient.java
+++ b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ReactiveAdminClient.java
@@ -212,17 +212,24 @@ private static Mono<Map<Integer, List<ConfigEntry>>> loadBrokersConfig(AdminClie
         .map(brokerId -> new ConfigResource(ConfigResource.Type.BROKER, Integer.toString(brokerId)))
         .collect(toList());
     return toMono(client.describeConfigs(resources).all())
-        // some kafka backends (like MSK serverless) do not support broker's configs retrieval,
-        // in that case InvalidRequestException will be thrown
-        .onErrorResume(InvalidRequestException.class, th -> {
-          log.trace("Error while getting broker {} configs", brokerIds, th);
-          return Mono.just(Map.of());
-        })
+        // some kafka backends don't support broker's configs retrieval,
+        // and throw various exceptions on describeConfigs() call
+        .onErrorResume(th -> th instanceof InvalidRequestException // MSK Serverless
+                || th instanceof UnknownTopicOrPartitionException, // Azure event hub
+            th -> {
+              log.trace("Error while getting configs for brokers {}", brokerIds, th);
+              return Mono.just(Map.of());
+            })
         // there are situations when kafka-ui user has no DESCRIBE_CONFIGS permission on cluster
         .onErrorResume(ClusterAuthorizationException.class, th -> {
           log.trace("AuthorizationException while getting configs for brokers {}", brokerIds, th);
           return Mono.just(Map.of());
         })
+        // catching all remaining exceptions, but logging on WARN level
+        .onErrorResume(th -> true, th -> {
+          log.warn("Unexpected error while getting configs for brokers {}", brokerIds, th);
+          return Mono.just(Map.of());
+        })
         .map(config -> config.entrySet().stream()
             .collect(toMap(
                 c -> Integer.valueOf(c.getKey().name()),


Test patch: