Repository: provectus/kafka-ui
Issue: ksql: schema vs data mismatch when structs 
Labels: type/bug
scope/backend
status/accepted

<!--

Don't forget to check for existing issues/discussions regarding your proposal. We might already have it.
https://github.com/provectus/kafka-ui/issues
https://github.com/provectus/kafka-ui/discussions

-->

<!--
Please follow the naming conventions for bugs:
<Feature/Area/Scope> :  <Compact, but specific problem summary> 
Avoid generic titles, like “Topics: incorrect layout of message sorting drop-down list”. Better use something like: “Topics: Message sorting drop-down list overlaps the "Submit" button”.

-->

**Describe the bug** (Actual behavior)
<!--(A clear and concise description of what the bug is.Use a list, if there is more than one problem)-->

When the stream schema contains structs, the headers seem to go down in the struct while the actual data in the table does not. The data seems correct and the headers seems bogus.

**Expected behavior**
<!--(A clear and concise description of what you expected to happen.)-->

Headers should match the data

**Set up**
<!--
WE MIGHT CLOSE THE ISSUE without further explanation IF YOU DON'T PROVIDE THIS INFORMATION.

How do you run the app? Please provide as much info as possible:
1. App version (docker image version or check commit hash in the top left corner in UI) v0.5.0
(
[027d9b4](https://github.com/provectus/kafka-ui/commit/027d9b4)
)
2. Helm chart version, if you use one
3. Any IAAC configs
-->


**Steps to Reproduce**
<!-- We'd like you to provide an example setup (via docker-compose, helm, etc.) 
to reproduce the problem, especially with a complex setups. -->

```ksql
CREATE OR REPLACE STREAM `raw_alfabet_applications` (
  `RefStr` VARCHAR KEY,
  `ClassName` VARCHAR,
  `Values` STRUCT<`Name` VARCHAR, `Short Name` VARCHAR, `Version` VARCHAR, `Description` VARCHAR, `Start Date` VARCHAR, `End Date` VARCHAR, `Object State` VARCHAR, `Status` VARCHAR, `Architecture Type` VARCHAR, `Development Type` VARCHAR, `Authentication` VARCHAR, `Alias` VARCHAR, `Recommendation` VARCHAR, `Strategic` VARCHAR, `PACE Governance` VARCHAR, `Cloud Migration Strategy` VARCHAR, `Compliance Regulation` VARCHAR, `Successor` VARCHAR, `Number of Users` VARCHAR, `Confidentiality` VARCHAR, `Integrity` VARCHAR, `Availability` VARCHAR, `Plan Start Date` VARCHAR, `Pilot Start Date` VARCHAR, `Production Start Date` VARCHAR, `Sunset Start Date` VARCHAR, `Retired Start Date` VARCHAR, `Lifecycle End Date` VARCHAR, `1 SW License Cost` VARCHAR, `2 SW Maintenance Cost` VARCHAR, `3 Infrastructure Cost` VARCHAR, `4 Service Cost` VARCHAR, `5 OpEx Other` VARCHAR, `Application Manager [Person]` VARCHAR, `Business Owner [Person]` VARCHAR, `IT Owner [Person]` VARCHAR, `Architect [Person]` VARCHAR, `Stakeholder [Person]` VARCHAR, `Business Owner [Organization]` VARCHAR, `IT Owner [Organization]` VARCHAR, `Operations [Organization]` VARCHAR, `Stakeholder [Organization]` VARCHAR>
) WITH (
  KAFKA_TOPIC='raw_alfabet_applications',
  KEY_FORMAT='KAFKA',
  VALUE_FORMAT='json'
);
```


**Screenshots**
<!--
(If applicable, add screenshots to help explain your problem)
-->

<img width="1629" alt="image" src="https://user-images.githubusercontent.com/659363/215692850-82210e3f-5d85-43a0-86f2-6883e94d5664.png">


**Additional context**
<!--
Add any other context about the problem here. E.g.: 
1. Are there any alternative scenarios (different data/methods/configuration/setup) you have tried? 
   Were they successfull or same issue occured? Please provide steps as well.
2. Related issues (if there are any).
3. Logs (if available)
4. Is there any serious impact or behaviour on the end-user because of this issue, that can be overlooked?
-->

If I was to look into this myself, where would I start looking?

Issue comments:
Hello there sdahlbac! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀
Hey, thanks for reporting, we'll take a look.

if you wanna take a look yourself, I'd start with this class: `com/provectus/kafka/ui/service/ksql/KsqlServiceV2.java:24`
@sdahlbac, thank you for the issue. Can you please confirm that it was fixed in `master`-tagged image (will be available in an hour)?

Pull request review comments:

Patch that resolved the issue:
diff --git a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ksql/response/ResponseParser.java b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ksql/response/ResponseParser.java
index 4781d159e7b..647e23a78ec 100644
--- a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ksql/response/ResponseParser.java
+++ b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/ksql/response/ResponseParser.java
@@ -3,14 +3,13 @@
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.json.JsonMapper;
 import com.fasterxml.jackson.databind.node.TextNode;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.Lists;
 import com.provectus.kafka.ui.exception.KsqlApiException;
 import com.provectus.kafka.ui.service.ksql.KsqlApiClient;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.List;
 import java.util.Optional;
-import java.util.stream.Collectors;
 import org.springframework.web.reactive.function.client.WebClientResponseException;
 
 public class ResponseParser {
@@ -24,11 +23,7 @@ public static Optional<KsqlApiClient.KsqlResponseTable> parseSelectResponse(Json
       return Optional.of(
           KsqlApiClient.KsqlResponseTable.builder()
               .header("Schema")
-              .columnNames(
-                  Arrays.stream(jsonNode.get("header").get("schema").asText().split(","))
-                      .map(String::trim)
-                      .collect(Collectors.toList())
-              )
+              .columnNames(parseSelectHeadersString(jsonNode.get("header").get("schema").asText()))
               .build());
     }
     if (arrayFieldNonEmpty(jsonNode, "row")) {
@@ -46,6 +41,34 @@ public static Optional<KsqlApiClient.KsqlResponseTable> parseSelectResponse(Json
     return Optional.empty();
   }
 
+  @VisibleForTesting
+  static List<String> parseSelectHeadersString(String str) {
+    List<String> headers = new ArrayList<>();
+    int structNesting = 0;
+    boolean quotes = false;
+    var headerBuilder = new StringBuilder();
+    for (char ch : str.toCharArray()) {
+      if (ch == '<') {
+        structNesting++;
+      } else if (ch == '>') {
+        structNesting--;
+      } else if (ch == '`') {
+        quotes = !quotes;
+      } else if (ch == ' ' && headerBuilder.isEmpty()) {
+        continue; //skipping leading & training whitespaces
+      } else if (ch == ',' && structNesting == 0 && !quotes) {
+        headers.add(headerBuilder.toString());
+        headerBuilder = new StringBuilder();
+        continue;
+      }
+      headerBuilder.append(ch);
+    }
+    if (!headerBuilder.isEmpty()) {
+      headers.add(headerBuilder.toString());
+    }
+    return headers;
+  }
+
   public static KsqlApiClient.KsqlResponseTable errorTableWithTextMsg(String errorText) {
     return KsqlApiClient.KsqlResponseTable.builder()
         .header("Execution error")


Test patch:
diff --git a/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/ksql/response/ResponseParserTest.java b/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/ksql/response/ResponseParserTest.java
new file mode 100644
index 00000000000..02552449336
--- /dev/null
+++ b/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/ksql/response/ResponseParserTest.java
@@ -0,0 +1,25 @@
+package com.provectus.kafka.ui.service.ksql.response;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import org.junit.jupiter.api.Test;
+
+class ResponseParserTest {
+
+  @Test
+  void parsesSelectHeaderIntoColumnNames() {
+    assertThat(ResponseParser.parseSelectHeadersString("`inQuotes` INT, notInQuotes INT"))
+        .containsExactly("`inQuotes` INT", "notInQuotes INT");
+
+    assertThat(ResponseParser.parseSelectHeadersString("`name with comma,` INT, name2 STRING"))
+        .containsExactly("`name with comma,` INT", "name2 STRING");
+
+    assertThat(ResponseParser.parseSelectHeadersString(
+        "`topLvl` INT, `struct` STRUCT<`nested1` STRING, anotherName STRUCT<nested2 INT>>"))
+        .containsExactly(
+            "`topLvl` INT",
+            "`struct` STRUCT<`nested1` STRING, anotherName STRUCT<nested2 INT>>"
+        );
+  }
+
+}