Repository: provectus/kafka-ui
Issue: Hide key magic bytes
Labels: type/bug
scope/backend

### Is your proposal related to a problem?
<!--
  Provide a clear and concise description of what the problem is.
  For example, "I'm always frustrated when..."
-->
I see the magic bytes when the key is deserialized by AVRO and I would like to hide them.
I saw the pull request following issue #515 but I noticed that the magic bytes are still displayed in the key, is this a desired behavior? Is it possible to hide them?

### Describe the solution you'd like
<!--
  Provide a clear and concise description of what you want to happen.
-->
Add a setting to hide magic bytes.

### Describe alternatives you've considered
<!--
  Let us know about other solutions you've tried or researched.
-->
Remove magic bytes.

Issue comments:
@Schiavi91. Thx for creating this issue. Could you please share your UI for Apache Kafka config, topic name, and schema names for value/key. 
I used docker-compose with this configuration:
```
kafka-ui:
    image: provectuslabs/kafka-ui:master
    hostname: kafka-ui
    container_name: kafka-ui
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - "8080:8080"
    restart: always
    environment:
      KAFKA_CLUSTERS_0_NAME: "local"
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "broker:29092"
      KAFKA_CLUSTERS_0_ZOOKEEPER: "zookeeper:2181"
      KAFKA_CLUSTERS_0_SCHEMAREGISTRY: "http://schema-registry:8081"
    networks:
      - my-net
```
The topic name is "my-topic"
The schema name of the key is "my-topic-key"
The schema name of the value is "my-topic-value"
Thanks for sharing.  You mentioned that there are magic key bytes in responses from sever. Could you please share your server response for this topic?
Sure, the server response is the following:
```
{
    "type": "MESSAGE",
    "message": {
        "partition": 0,
        "offset": 0,
        "timestamp": "2021-09-21T13:13:17.837Z",
        "timestampType": "CREATE_TIME",
        "key": "\u0000\u0000\u0000\u0000\u0001\u0006asd",
        "headers": {},
        "content": "{\"number\":123}",
        "keyFormat": "UNKNOWN",
        "valueFormat": "AVRO",
        "keySize": 9,
        "valueSize": 7,
        "keySchemaId": null,
        "valueSchemaId": "2",
        "headersSize": 0
    },
    "phase": null,
    "consuming": null
}
```
While in the frontend I see this:
![image](https://user-images.githubusercontent.com/38749012/134178031-5ea54953-b683-4607-a469-21665a5a662a.png)
@Schiavi91, currently we are finding deserializer(both for key and value) on the first topic message read. After that, it is cached and will be used for all messages in the topic.
In your case for some reason we assigned default (String based) deserializer to topic's key. Maybe key schema was added after first message was written? 
Can you please try to restart kafka-ui and try to read topic again?
Hi @iliax, thanks for the reply. I first created the schemas for key and value then I generated the first message, otherwise I would not have known the id to use to serialize the key.
I tried to restart kafka-ui and read topic again but the result is the same.
@Schiavi91 can you please share schema you created for key?
"string"

The call to the schema registry is the following:
```
curl --location --request POST 'http://localhost:8081/subjects/my-topic-key/versions/' --header 'Accept: application/json' --header 'Content-Type: application/json' --data-raw '{ "schema": "\"string\""}'
```
@Schiavi91  we found and fixed a bug with avro deserilization with primitive type schemas. Can you please check?
@iliax I just checked and now it works.
Given the change made, I noticed two other things that might make sense to check:
1. in the "key" field there are the quotation marks that could be removed (inside the red circle)
2. in the message detail window there is a colon symbol before the key (inside the blue circle), is this the desired behavior? Or should it be removed?

![image](https://user-images.githubusercontent.com/38749012/134548680-f0851856-69e4-48a2-918d-f7dc9f37d0d2.png)
@Schiavi91, we return value in json representation, this is why quotes are present. As for colon - I will talk with UI devs, to fix this. 
@Schiavi91 I discussed display with UI devs - this is expected for UI component we currently use. But I agree that it looks strange, so we will discuss how we can improve it in next release (0.3)
Ok thanks @iliax, i think the issue can be closed.

Pull request review comments:

Patch that resolved the issue:
diff --git a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serde/schemaregistry/AvroMessageFormatter.java b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serde/schemaregistry/AvroMessageFormatter.java
index 817ade0fb51..e69d522ea03 100644
--- a/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serde/schemaregistry/AvroMessageFormatter.java
+++ b/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serde/schemaregistry/AvroMessageFormatter.java
@@ -4,7 +4,6 @@
 import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
 import io.confluent.kafka.serializers.KafkaAvroDeserializer;
 import lombok.SneakyThrows;
-import org.apache.avro.generic.GenericRecord;
 
 public class AvroMessageFormatter implements MessageFormatter {
   private final KafkaAvroDeserializer avroDeserializer;
@@ -16,8 +15,10 @@ public AvroMessageFormatter(SchemaRegistryClient client) {
   @Override
   @SneakyThrows
   public String format(String topic, byte[] value) {
-    GenericRecord avroRecord = (GenericRecord) avroDeserializer.deserialize(topic, value);
-    byte[] jsonBytes = AvroSchemaUtils.toJson(avroRecord);
+    // deserialized will have type, that depends on schema type (record or primitive),
+    // AvroSchemaUtils.toJson(...) method will take it into account
+    Object deserialized = avroDeserializer.deserialize(topic, value);
+    byte[] jsonBytes = AvroSchemaUtils.toJson(deserialized);
     return new String(jsonBytes);
   }
 


Test patch:
diff --git a/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/SendAndReadTests.java b/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/SendAndReadTests.java
index 7bd16af2725..3b578a6a892 100644
--- a/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/SendAndReadTests.java
+++ b/kafka-ui-api/src/test/java/com/provectus/kafka/ui/service/SendAndReadTests.java
@@ -65,6 +65,13 @@ public class SendAndReadTests extends AbstractBaseTest {
           + "}"
   );
 
+  private static final AvroSchema AVRO_SCHEMA_PRIMITIVE_STRING =
+      new AvroSchema("{ \"type\": \"string\" }");
+
+  private static final AvroSchema AVRO_SCHEMA_PRIMITIVE_INT =
+      new AvroSchema("{ \"type\": \"int\" }");
+
+
   private static final String AVRO_SCHEMA_1_JSON_RECORD
       = "{ \"field1\":\"testStr\", \"field2\": 123 }";
 
@@ -187,6 +194,22 @@ void noSchemaValueIsNull() {
         });
   }
 
+  @Test
+  void primitiveAvroSchemas() {
+    new SendAndReadSpec()
+        .withKeySchema(AVRO_SCHEMA_PRIMITIVE_STRING)
+        .withValueSchema(AVRO_SCHEMA_PRIMITIVE_INT)
+        .withMsgToSend(
+            new CreateTopicMessage()
+                .key("\"some string\"")
+                .content("123")
+        )
+        .doAssert(polled -> {
+          assertThat(polled.getKey()).isEqualTo("\"some string\"");
+          assertThat(polled.getContent()).isEqualTo("123");
+        });
+  }
+
   @Test
   void nonNullableKvWithAvroSchema() {
     new SendAndReadSpec()