Repository: apache/pinot
Pull request #15273: feat(KafkaConfluentSchemaRegistryJsonMessageDecoder): added the JSON confluent schema decoder
Author: login: r-reis
Labels: name: release-notes
description: Referenced by PRs that need attention when compiling the next release notes
name: configuration
description: Config changes (addition/deletion/change in behavior)
name: ingestion
description: Related to data ingestion pipeline
name: real-time
description: Related to realtime table ingestion and serving
Created: 2025-03-14T13:51:48Z | Merged: 2025-03-22T18:55:38Z

Related to this [issue](https://github.com/apache/pinot/issues/15262)

The main change was the inclusion of confluent [JSONSchemaDeserializer](https://github.com/confluentinc/schema-registry/blob/master/json-schema-serializer/src/main/java/io/confluent/kafka/serializers/json/KafkaJsonSchemaDeserializer.java) to the input-format plugin.

Follow the instruction on the issue, but now we have a new decoder.class.name "org.apache.pinot.plugin.inputformat.json.confluent.KafkaConfluentSchemaRegistryJsonMessageDecoder"

Example of table config:

```
{
  "tableName": "topic_1",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "created_at",
    "timeType": "MILLISECONDS",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "stream.kafka.metadata.populate" : "true",
      "streamType": "kafka",
      "stream.kafka.decoder.prop.format": "JSON",
      "stream.kafka.consumer.type": "low-level",
      "stream.kafka.topic.name": "topic_1",

      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.confluent.KafkaConfluentSchemaRegistryJsonMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory",
     	
      "stream.kafka.schema.registry.url": "http://localhost:8081",
      "stream.kafka.decoder.prop.schema.registry.rest.url": "http://localhost:8081",		
      "stream.kafka.broker.list": "localhost:9092",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest",		
      "key.serializer": "shaded.org.apache.kafka.connect.storage.StringDeserializer",
      "value.serializer": "shaded.org.apache.kafka.connect.storage.StringDeserializer"
    }
  },
	"ingestionConfig": {
		"continueOnError": true
	},
  "metadata": {
    "customConfigs": {}
  }
}
```

Handling all exceptions and errors is the same as AvroSchemaDecoder and ProtoBufSchemaDecoder.


Diff:
diff --git a/pinot-distribution/pinot-assembly.xml b/pinot-distribution/pinot-assembly.xml
index 04a88924c46a..078205f21315 100644
--- a/pinot-distribution/pinot-assembly.xml
+++ b/pinot-distribution/pinot-assembly.xml
@@ -189,6 +189,12 @@
       </source>
       <destName>plugins/pinot-input-format/pinot-confluent-protobuf/pinot-confluent-protobuf-${project.version}-shaded.jar</destName>
     </file>
+    <file>
+      <source>
+        ${pinot.root}/pinot-plugins/pinot-input-format/pinot-confluent-json/target/pinot-confluent-json-${project.version}-shaded.jar
+      </source>
+      <destName>plugins/pinot-input-format/pinot-confluent-json/pinot-confluent-json-${project.version}-shaded.jar</destName>
+    </file>
     <!-- End Include Pinot Input Format Plugins-->
     <!-- Start Include Pinot Minion Tasks Plugins-->
     <file>
diff --git a/pinot-plugins/pinot-input-format/pinot-confluent-json/pom.xml b/pinot-plugins/pinot-input-format/pinot-confluent-json/pom.xml
new file mode 100644
index 000000000000..416c98cecb49
--- /dev/null
+++ b/pinot-plugins/pinot-input-format/pinot-confluent-json/pom.xml
@@ -0,0 +1,99 @@
+<?xml version="1.0"?>
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <artifactId>pinot-input-format</artifactId>
+    <groupId>org.apache.pinot</groupId>
+    <version>1.4.0-SNAPSHOT</version>
+  </parent>
+
+  <artifactId>pinot-confluent-json</artifactId>
+  <name>Pinot Confluent Json</name>
+  <url>https://pinot.apache.org/</url>
+  <properties>
+    <pinot.root>${basedir}/../../..</pinot.root>
+    <shade.phase.prop>package</shade.phase.prop>
+  </properties>
+  <repositories>
+    <repository>
+      <id>confluent</id>
+      <url>https://packages.confluent.io/maven/</url>
+    </repository>
+  </repositories>
+  <dependencies>
+    <dependency>
+      <groupId>org.apache.pinot</groupId>
+      <artifactId>pinot-json</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.kafka</groupId>
+      <artifactId>kafka-clients</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.confluent</groupId>
+      <artifactId>kafka-schema-registry-client</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>io.confluent</groupId>
+      <artifactId>kafka-json-schema-serializer</artifactId>
+      <exclusions>
+        <exclusion>
+          <groupId>commons-logging</groupId>
+          <artifactId>commons-logging</artifactId>
+        </exclusion>
+        <exclusion>
+          <groupId>org.jetbrains.kotlin</groupId>
+          <artifactId>kotlin-scripting-compiler-embeddable</artifactId>
+        </exclusion>
+      </exclusions>
+    </dependency>
+    <dependency>
+      <groupId>org.testcontainers</groupId>
+      <artifactId>testcontainers</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>org.testcontainers</groupId>
+      <artifactId>kafka</artifactId>
+    </dependency>
+    <dependency>
+      <groupId>com.google.protobuf</groupId>
+      <artifactId>protobuf-java</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.scala-lang</groupId>
+      <artifactId>scala-library</artifactId>
+      <version>2.13.16</version>
+      <scope>test</scope>
+    </dependency>
+  </dependencies>
+
+  <profiles>
+    <profile>
+      <id>pinot-fastdev</id>
+      <properties>
+        <shade.phase.prop>none</shade.phase.prop>
+      </properties>
+    </profile>
+  </profiles>
+</project>
diff --git a/pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java b/pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java
new file mode 100644
index 000000000000..7abde436efda
--- /dev/null
+++ b/pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.rest.RestService;
+import io.confluent.kafka.schemaregistry.json.JsonSchemaProvider;
+import io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Consumer;
+import javax.annotation.Nullable;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.SslConfigs;
+import org.apache.kafka.common.config.types.Password;
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.security.ssl.DefaultSslEngineFactory;
+import org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordExtractor;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Preconditions.checkState;
+
+
+public class KafkaConfluentSchemaRegistryJsonMessageDecoder implements StreamMessageDecoder<byte[]> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(KafkaConfluentSchemaRegistryJsonMessageDecoder.class);
+  private static final String SCHEMA_REGISTRY_REST_URL = "schema.registry.rest.url";
+  private static final String SCHEMA_REGISTRY_OPTS_PREFIX = "schema.registry.";
+  public static final String CACHED_SCHEMA_MAP_CAPACITY = "cached.schema.map.capacity";
+  public static final String DEFAULT_CACHED_SCHEMA_MAP_CAPACITY = "1000";
+  private KafkaJsonSchemaDeserializer _deserializer;
+  private RecordExtractor<Map<String, Object>> _jsonRecordExtractor;
+  private String _topicName;
+  @Nullable
+  private final Consumer<Map<String, Object>> _onMessage;
+
+  /**
+   * Creates a new instance of this decoder. This constructor with no argument is usually called by reflection
+   */
+  public KafkaConfluentSchemaRegistryJsonMessageDecoder() {
+    _onMessage = null;
+  }
+
+  /**
+   * Creates a new instance of this decoder. This constructor with a Consumer argument is used by test to be able to
+   * analyze the received message.
+   */
+  @VisibleForTesting
+  KafkaConfluentSchemaRegistryJsonMessageDecoder(@Nullable Consumer<Map<String, Object>> onMessage) {
+    _onMessage = onMessage;
+  }
+
+  private RestService createRestService(String schemaRegistryUrl, Map<String, String> configs) {
+    RestService restService = new RestService(schemaRegistryUrl);
+
+    ConfigDef configDef = new ConfigDef();
+    SslConfigs.addClientSslSupport(configDef);
+    Map<String, ConfigDef.ConfigKey> configKeyMap = configDef.configKeys();
+    Map<String, Object> sslConfigs = new HashMap<>();
+    for (String key : configs.keySet()) {
+      if (!key.equals(SCHEMA_REGISTRY_REST_URL) && key.startsWith(SCHEMA_REGISTRY_OPTS_PREFIX)) {
+        String value = configs.get(key);
+        String schemaRegistryOptKey = key.substring(SCHEMA_REGISTRY_OPTS_PREFIX.length());
+
+        if (configKeyMap.containsKey(schemaRegistryOptKey)) {
+          if (configKeyMap.get(schemaRegistryOptKey).type == ConfigDef.Type.PASSWORD) {
+            sslConfigs.put(schemaRegistryOptKey, new Password(value));
+          } else {
+            sslConfigs.put(schemaRegistryOptKey, value);
+          }
+        }
+      }
+    }
+
+    if (!sslConfigs.isEmpty()) {
+      DefaultSslEngineFactory sslFactory = new DefaultSslEngineFactory();
+      sslFactory.configure(sslConfigs);
+      restService.setSslSocketFactory(sslFactory.sslContext().getSocketFactory());
+    }
+    return restService;
+  }
+
+  @Override
+  public void init(Map<String, String> props, Set<String> fieldsToRead, String topicName)
+      throws Exception {
+    checkState(props.containsKey(SCHEMA_REGISTRY_REST_URL), "Missing required property '%s'", SCHEMA_REGISTRY_REST_URL);
+    String schemaRegistryUrl = props.get(SCHEMA_REGISTRY_REST_URL);
+    JsonSchemaProvider jsonSchemaProvider = new JsonSchemaProvider();
+    int identityMapCapacity = Integer.parseInt(
+            props.getOrDefault(CACHED_SCHEMA_MAP_CAPACITY, DEFAULT_CACHED_SCHEMA_MAP_CAPACITY));
+    SchemaRegistryClient schemaRegistryClient =
+        new CachedSchemaRegistryClient(createRestService(schemaRegistryUrl, props), identityMapCapacity,
+                Collections.singletonList(jsonSchemaProvider), props, null);
+
+    _deserializer = new KafkaJsonSchemaDeserializer(schemaRegistryClient);
+    Preconditions.checkNotNull(topicName, "Topic must be provided");
+    _topicName = topicName;
+    _jsonRecordExtractor = PluginManager.get().createInstance(JSONRecordExtractor.class.getName());
+    _jsonRecordExtractor.init(fieldsToRead, null);
+  }
+
+  @Override
+  public GenericRow decode(byte[] payload, GenericRow destination) {
+    try {
+      JsonNode jsonRecord = (JsonNode) _deserializer.deserialize(_topicName, payload);
+      Map<String, Object> from = JsonUtils.jsonNodeToMap(jsonRecord);
+      return _jsonRecordExtractor.extract(from, destination);
+    } catch (RuntimeException | IOException e) {
+      ignoreOrRethrowException(e);
+      return null;
+    }
+  }
+
+  @Override
+  public GenericRow decode(byte[] payload, int offset, int length, GenericRow destination) {
+    return decode(Arrays.copyOfRange(payload, offset, offset + length), destination);
+  }
+
+  /**
+   * This method handles specific serialisation exceptions. If the exception cannot be ignored the method
+   * re-throws the exception.
+   *
+   * @param e exception to handle
+   */
+  private void ignoreOrRethrowException(RuntimeException e) {
+    if (isUnknownMagicByte(e) || isUnknownMagicByte(e.getCause())) {
+      // Do nothing, the message is not an JSON message and can't be decoded
+      LOGGER.error("Caught exception while decoding row in topic {}, discarding row", _topicName, e);
+      return;
+    }
+    throw e;
+  }
+
+  private void ignoreOrRethrowException(Exception e) {
+    if (isUnknownMagicByte(e) || (e.getCause() != null && isUnknownMagicByte(e.getCause()))) {
+      LOGGER.error("Caught IOexception while mapping jsonNode in topic {}, discarding row", _topicName, e);
+      return;
+    }
+    throw new RuntimeException(e);
+  }
+
+  private boolean isUnknownMagicByte(Throwable e) {
+    return e != null && e instanceof SerializationException && e.getMessage() != null && e.getMessage().toLowerCase()
+        .contains("unknown magic byte");
+  }
+}
diff --git a/pinot-plugins/pinot-input-format/pinot-confluent-json/src/test/java/org/apache/pinot/plugin/inputformat/json/confluent/JsonConfluentSchemaTest.java b/pinot-plugins/pinot-input-format/pinot-confluent-json/src/test/java/org/apache/pinot/plugin/inputformat/json/confluent/JsonConfluentSchemaTest.java
new file mode 100644
index 000000000000..80937b442a25
--- /dev/null
+++ b/pinot-plugins/pinot-input-format/pinot-confluent-json/src/test/java/org/apache/pinot/plugin/inputformat/json/confluent/JsonConfluentSchemaTest.java
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent;
+
+import io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializerConfig;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.UUID;
+import java.util.function.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.pinot.plugin.inputformat.json.confluent.kafka.schemaregistry.SchemaRegistryStarter;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+public class JsonConfluentSchemaTest {
+  public static final String TOPIC_JSON = "test_topic_json";
+  SchemaRegistryStarter.KafkaSchemaRegistryInstance _schemaRegistry;
+  private Producer<byte[], Map<String, Object>> _jsonProducer;
+
+  @BeforeClass
+  public void setup() {
+    _schemaRegistry = SchemaRegistryStarter.startLocalInstance(9093);
+
+    Properties jsonProducerProps = new Properties();
+    jsonProducerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
+        _schemaRegistry._kafkaContainer.getBootstrapServers());
+    jsonProducerProps.put(KafkaJsonSchemaSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, _schemaRegistry.getUrl());
+    jsonProducerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
+        "org.apache.kafka.common.serialization.ByteArraySerializer");
+    jsonProducerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
+        "io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer");
+    _jsonProducer = new KafkaProducer<>(jsonProducerProps);
+  }
+
+  @Test
+  public void testSamplePinotConsumer()
+      throws Exception {
+    int numRecords = 10;
+    List<Map<String, Object>> recordList = new ArrayList<>();
+    for (int i = 0; i < numRecords; i++) {
+      Map<String, Object> sampleRecord = new HashMap<>();
+      sampleRecord.put("name", UUID.randomUUID().toString());
+      sampleRecord.put("email", UUID.randomUUID().toString());
+      List<String> friends = new ArrayList<>();
+      friends.add(UUID.randomUUID().toString());
+      friends.add(UUID.randomUUID().toString());
+      sampleRecord.put("friends", friends);
+      sampleRecord.put("id", i);
+      if (i % 2 == 0) {
+        sampleRecord.put("optionalField", UUID.randomUUID().toString());
+      }
+
+      _jsonProducer.send(new ProducerRecord<>(TOPIC_JSON, sampleRecord));
+      recordList.add(sampleRecord);
+    }
+
+    Properties consumerProps = new Properties();
+    consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, _schemaRegistry._kafkaContainer.getBootstrapServers());
+    consumerProps.put(KafkaJsonSchemaSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, _schemaRegistry.getUrl());
+    consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
+        "org.apache.kafka.common.serialization.ByteArrayDeserializer");
+    consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
+        "org.apache.kafka.common.serialization.ByteArrayDeserializer");
+    consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+    consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, "foo_bar");
+    KafkaConsumer<byte[], byte[]> kafkaConsumer = new KafkaConsumer<>(consumerProps);
+    kafkaConsumer.subscribe(Collections.singletonList(TOPIC_JSON));
+    ConsumerRecords<byte[], byte[]> consumerRecords = kafkaConsumer.poll(Duration.ofMillis(1000));
+    Iterator<ConsumerRecord<byte[], byte[]>> iter = consumerRecords.iterator();
+
+    Consumer<Map<String, Object>> onMessage = message -> {
+      String optionalField = (String) message.get("optionalField");
+      Assert.assertNull(optionalField, "Received json have been rewritten");
+    };
+
+    KafkaConfluentSchemaRegistryJsonMessageDecoder decoder =
+        new KafkaConfluentSchemaRegistryJsonMessageDecoder(onMessage);
+    Map<String, String> decoderProps = new HashMap<>();
+    decoderProps.put("schema.registry.rest.url", _schemaRegistry.getUrl());
+    decoder.init(decoderProps, null, TOPIC_JSON);
+    GenericRow reuse = new GenericRow();
+    List<GenericRow> result = new ArrayList<>();
+    while (iter.hasNext()) {
+      byte[] arr = iter.next().value();
+      decoder.decode(arr, reuse);
+      result.add(reuse.copy());
+      reuse.clear();
+    }
+
+    Assert.assertEquals(result.size(), numRecords);
+
+    for (int i = 0; i < numRecords; i++) {
+      Map<String, Object> originalValue = recordList.get(i);
+      GenericRow decodedValue = result.get(i);
+
+      Assert.assertEquals(decodedValue.getValue("name"),
+          originalValue.get("name"), "Unexpected 'name' value");
+      Assert.assertEquals(decodedValue.getValue("id"),
+          originalValue.get("id"), "Unexpected 'id' value");
+      Assert.assertEquals(decodedValue.getValue("email"),
+          originalValue.get("email"), "Unexpected 'email' value");
+
+      Object[] expectedFriends = ((List<String>) originalValue.get("friends")).toArray(new String[0]);
+      Assert.assertEquals(decodedValue.getValue("friends"),
+          expectedFriends, "Unexpected 'friends' value");
+      String expectedOptionalField = i % 2 == 0 ? (String) originalValue.get("optionalField") : null;
+      Assert.assertEquals(decodedValue.getValue("optionalField"),
+          expectedOptionalField, "Unexpected 'optionalField' value");
+    }
+  }
+
+  @AfterClass
+  public void tearDown() {
+    _schemaRegistry.stop();
+  }
+}
diff --git a/pinot-plugins/pinot-input-format/pinot-confluent-json/src/test/java/org/apache/pinot/plugin/inputformat/json/confluent/kafka/schemaregistry/SchemaRegistryStarter.java b/pinot-plugins/pinot-input-format/pinot-confluent-json/src/test/java/org/apache/pinot/plugin/inputformat/json/confluent/kafka/schemaregistry/SchemaRegistryStarter.java
new file mode 100644
index 000000000000..f38d94f790e1
--- /dev/null
+++ b/pinot-plugins/pinot-input-format/pinot-confluent-json/src/test/java/org/apache/pinot/plugin/inputformat/json/confluent/kafka/schemaregistry/SchemaRegistryStarter.java
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent.kafka.schemaregistry;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.KafkaContainer;
+import org.testcontainers.containers.Network;
+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+
+public class SchemaRegistryStarter {
+  public static final int DEFAULT_PORT = 8081;
+  private static final String CONFLUENT_PLATFORM_VERSION = "7.2.0";
+  private static final DockerImageName KAFKA_DOCKER_IMAGE_NAME =
+      DockerImageName.parse("confluentinc/cp-kafka:" + CONFLUENT_PLATFORM_VERSION);
+  private static final DockerImageName SCHEMA_REGISTRY_DOCKER_IMAGE_NAME =
+      DockerImageName.parse("confluentinc/cp-schema-registry:" + CONFLUENT_PLATFORM_VERSION);
+  private static final Logger LOGGER = LoggerFactory.getLogger(SchemaRegistryStarter.class);
+
+  private SchemaRegistryStarter() {
+  }
+
+  public static KafkaSchemaRegistryInstance startLocalInstance(int port) {
+    KafkaSchemaRegistryInstance kafkaSchemaRegistry = new KafkaSchemaRegistryInstance(port);
+    kafkaSchemaRegistry.start();
+    return kafkaSchemaRegistry;
+  }
+
+  public static class KafkaSchemaRegistryInstance {
+    private final int _port;
+    public KafkaContainer _kafkaContainer;
+    private Network _network;
+    private GenericContainer _schemaRegistryContainer;
+
+    private KafkaSchemaRegistryInstance(int port) {
+      _port = port;
+    }
+
+    public String getUrl() {
+      return "http://" + _schemaRegistryContainer.getHost() + ":" + _schemaRegistryContainer.getMappedPort(_port);
+    }
+
+    public void start() {
+      LOGGER.info("Starting schema registry");
+      if (_kafkaContainer != null || _schemaRegistryContainer != null) {
+        throw new IllegalStateException("Schema registry is already running");
+      }
+
+      _network = Network.newNetwork();
+
+      _kafkaContainer = new KafkaContainer(KAFKA_DOCKER_IMAGE_NAME).withNetwork(_network).withNetworkAliases("kafka")
+          .withCreateContainerCmdModifier(it -> it.withHostName("kafka")).waitingFor(Wait.forListeningPort());
+      _kafkaContainer.start();
+
+      Map<String, String> schemaRegistryProps = new HashMap<>();
+      schemaRegistryProps.put("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", "kafka:9092");
+      schemaRegistryProps.put("SCHEMA_REGISTRY_HOST_NAME", "schemaregistry");
+      schemaRegistryProps.put("SCHEMA_REGISTRY_LISTENERS", "http://0.0.0.0:" + _port);
+      schemaRegistryProps.put("SCHEMA_REGISTRY_DEBUG", "true");
+      _schemaRegistryContainer =
+          new GenericContainer(SCHEMA_REGISTRY_DOCKER_IMAGE_NAME).dependsOn(_kafkaContainer).withNetwork(_network)
+              .withNetworkAliases("schemaregistry").withEnv(schemaRegistryProps).withExposedPorts(_port)
+              .waitingFor(Wait.forListeningPort());
+      _schemaRegistryContainer.start();
+    }
+
+    public void stop() {
+      LOGGER.info("Stopping schema registry");
+      if (_schemaRegistryContainer != null) {
+        _schemaRegistryContainer.stop();
+        _schemaRegistryContainer = null;
+      }
+
+      if (_kafkaContainer != null) {
+        _kafkaContainer.stop();
+        _kafkaContainer = null;
+      }
+
+      if (_network != null) {
+        _network.close();
+      }
+    }
+  }
+}
diff --git a/pinot-plugins/pinot-input-format/pom.xml b/pinot-plugins/pinot-input-format/pom.xml
index 6f39b655199f..b942d8e2ceeb 100644
--- a/pinot-plugins/pinot-input-format/pom.xml
+++ b/pinot-plugins/pinot-input-format/pom.xml
@@ -41,6 +41,7 @@
     <module>pinot-avro-base</module>
     <module>pinot-clp-log</module>
     <module>pinot-confluent-avro</module>
+    <module>pinot-confluent-json</module>
     <module>pinot-confluent-protobuf</module>
     <module>pinot-orc</module>
     <module>pinot-json</module>
diff --git a/pom.xml b/pom.xml
index 91cac51c305e..c857b8d1b4f1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -585,6 +585,11 @@
         <artifactId>pinot-confluent-protobuf</artifactId>
         <version>${project.version}</version>
       </dependency>
+      <dependency>
+        <groupId>org.apache.pinot</groupId>
+        <artifactId>pinot-confluent-json</artifactId>
+        <version>${project.version}</version>
+      </dependency>
       <dependency>
         <groupId>org.apache.pinot</groupId>
         <artifactId>pinot-orc</artifactId>
@@ -1705,6 +1710,11 @@
         <artifactId>kafka-protobuf-serializer</artifactId>
         <version>${confluent.version}</version>
       </dependency>
+      <dependency>
+        <groupId>io.confluent</groupId>
+        <artifactId>kafka-json-schema-serializer</artifactId>
+        <version>${confluent.version}</version>
+      </dependency>
 
       <dependency>
         <groupId>org.apache.pulsar</groupId>


Reviews:
user: login: rajagopr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T23:20:13Z
user: login: r-reis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T22:33:10Z
user: login: r-reis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T22:33:36Z
user: login: r-reis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T22:35:17Z
user: login: mayankshriv
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-22T18:55:31Z

Review comments:
diff_hunk: @@ -1929,6 +1939,11 @@
         <artifactId>kotlin-reflect</artifactId>
         <version>${kotlin.stdlib.version}</version>
       </dependency>
+      <dependency>
path: pom.xml
user: login: rajagopr
body: Why do we need this plugin?
created_at: 2025-03-17T23:07:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.base.Preconditions;
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.rest.RestService;
+import io.confluent.kafka.schemaregistry.json.JsonSchemaProvider;
+import io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.SslConfigs;
+import org.apache.kafka.common.config.types.Password;
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.security.ssl.DefaultSslEngineFactory;
+import org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordExtractor;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Preconditions.checkState;
+
+
+/**
+ * Decodes avro messages with confluent schema registry.
+ * First byte is MAGIC = 0, second 4 bytes are the schema id, the remainder is the value.
+ * NOTE: Do not use schema in the implementation, as schema will be removed from the params
+ */
+public class KafkaConfluentSchemaRegistryJsonMessageDecoder implements StreamMessageDecoder<byte[]> {
path: pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java
user: login: rajagopr
body: Can we also add an accompanying test? Similar to the one for `proto`?
created_at: 2025-03-17T23:19:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.base.Preconditions;
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.rest.RestService;
+import io.confluent.kafka.schemaregistry.json.JsonSchemaProvider;
+import io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.SslConfigs;
+import org.apache.kafka.common.config.types.Password;
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.security.ssl.DefaultSslEngineFactory;
+import org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordExtractor;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Preconditions.checkState;
+
+
+/**
+ * Decodes avro messages with confluent schema registry.
+ * First byte is MAGIC = 0, second 4 bytes are the schema id, the remainder is the value.
+ * NOTE: Do not use schema in the implementation, as schema will be removed from the params
+ */
+public class KafkaConfluentSchemaRegistryJsonMessageDecoder implements StreamMessageDecoder<byte[]> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(KafkaConfluentSchemaRegistryJsonMessageDecoder.class);
+  private static final String SCHEMA_REGISTRY_REST_URL = "schema.registry.rest.url";
+  private static final String SCHEMA_REGISTRY_OPTS_PREFIX = "schema.registry.";
+  public static final String CACHED_SCHEMA_MAP_CAPACITY = "cached.schema.map.capacity";
+  public static final String DEFAULT_CACHED_SCHEMA_MAP_CAPACITY = "1000";
+  private KafkaJsonSchemaDeserializer _deserializer;
+  private RecordExtractor<Map<String, Object>> _jsonRecordExtractor;
+  private String _topicName;
+
+  public RestService createRestService(String schemaRegistryUrl, Map<String, String> configs) {
+    RestService restService = new RestService(schemaRegistryUrl);
+
+    ConfigDef configDef = new ConfigDef();
+    SslConfigs.addClientSslSupport(configDef);
+    Map<String, ConfigDef.ConfigKey> configKeyMap = configDef.configKeys();
+    Map<String, Object> sslConfigs = new HashMap<>();
+    for (String key : configs.keySet()) {
+      if (!key.equals(SCHEMA_REGISTRY_REST_URL) && key.startsWith(SCHEMA_REGISTRY_OPTS_PREFIX)) {
+        String value = configs.get(key);
+        String schemaRegistryOptKey = key.substring(SCHEMA_REGISTRY_OPTS_PREFIX.length());
+
+        if (configKeyMap.containsKey(schemaRegistryOptKey)) {
+          if (configKeyMap.get(schemaRegistryOptKey).type == ConfigDef.Type.PASSWORD) {
+            sslConfigs.put(schemaRegistryOptKey, new Password(value));
+          } else {
+            sslConfigs.put(schemaRegistryOptKey, value);
+          }
+        }
+      }
+    }
+
+    if (!sslConfigs.isEmpty()) {
+      DefaultSslEngineFactory sslFactory = new DefaultSslEngineFactory();
+      sslFactory.configure(sslConfigs);
+      restService.setSslSocketFactory(sslFactory.sslContext().getSocketFactory());
+    }
+    return restService;
+  }
+
+  @Override
+  public void init(Map<String, String> props, Set<String> fieldsToRead, String topicName)
+      throws Exception {
+    checkState(props.containsKey(SCHEMA_REGISTRY_REST_URL), "Missing required property '%s'", SCHEMA_REGISTRY_REST_URL);
+    String schemaRegistryUrl = props.get(SCHEMA_REGISTRY_REST_URL);
+    JsonSchemaProvider jsonSchemaProvider = new JsonSchemaProvider();
+    int identityMapCapacity = Integer.parseInt(
+            props.getOrDefault(CACHED_SCHEMA_MAP_CAPACITY, DEFAULT_CACHED_SCHEMA_MAP_CAPACITY));
+    SchemaRegistryClient schemaRegistryClient =
+        new CachedSchemaRegistryClient(createRestService(schemaRegistryUrl, props), identityMapCapacity,
+                Collections.singletonList(jsonSchemaProvider), props, null);
+
+    _deserializer = new KafkaJsonSchemaDeserializer(schemaRegistryClient);
+    Preconditions.checkNotNull(topicName, "Topic must be provided");
+    _topicName = topicName;
+    _jsonRecordExtractor = PluginManager.get().createInstance(JSONRecordExtractor.class.getName());
+    _jsonRecordExtractor.init(fieldsToRead, null);
+  }
+
+  @Override
+  public GenericRow decode(byte[] payload, GenericRow destination) {
+    try {
+      JsonNode jsonRecord = (JsonNode) _deserializer.deserialize(_topicName, payload);
+      Map<String, Object> from = JsonUtils.jsonNodeToMap(jsonRecord);
+      return _jsonRecordExtractor.extract(from, destination);
+    } catch (RuntimeException e) {
+      ignoreOrRethrowException(e);
+      return null;
+    } catch (Exception e) {
+      LOGGER.error("Caught exception while decoding row, discarding row. Payload is {}", new String(payload), e);
+      return null;
+    }
path: pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java
user: login: rajagopr
body: The other two handlers seem to propagate the exception upwards and not handle it in a generic exception block like this. Should we keep the behavior consistent for all three decoders?
created_at: 2025-03-17T23:20:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.base.Preconditions;
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.rest.RestService;
+import io.confluent.kafka.schemaregistry.json.JsonSchemaProvider;
+import io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.SslConfigs;
+import org.apache.kafka.common.config.types.Password;
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.security.ssl.DefaultSslEngineFactory;
+import org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordExtractor;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Preconditions.checkState;
+
+
+/**
+ * Decodes avro messages with confluent schema registry.
+ * First byte is MAGIC = 0, second 4 bytes are the schema id, the remainder is the value.
+ * NOTE: Do not use schema in the implementation, as schema will be removed from the params
+ */
+public class KafkaConfluentSchemaRegistryJsonMessageDecoder implements StreamMessageDecoder<byte[]> {
path: pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java
user: login: r-reis
body: Done!
created_at: 2025-03-18T22:33:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1929,6 +1939,11 @@
         <artifactId>kotlin-reflect</artifactId>
         <version>${kotlin.stdlib.version}</version>
       </dependency>
+      <dependency>
path: pom.xml
user: login: r-reis
body: It was a leftover from previous tests.
Fixed and ty.
created_at: 2025-03-18T22:33:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,153 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.json.confluent;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.google.common.base.Preconditions;
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.rest.RestService;
+import io.confluent.kafka.schemaregistry.json.JsonSchemaProvider;
+import io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.SslConfigs;
+import org.apache.kafka.common.config.types.Password;
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.security.ssl.DefaultSslEngineFactory;
+import org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordExtractor;
+import org.apache.pinot.spi.plugin.PluginManager;
+import org.apache.pinot.spi.stream.StreamMessageDecoder;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static com.google.common.base.Preconditions.checkState;
+
+
+/**
+ * Decodes avro messages with confluent schema registry.
+ * First byte is MAGIC = 0, second 4 bytes are the schema id, the remainder is the value.
+ * NOTE: Do not use schema in the implementation, as schema will be removed from the params
+ */
+public class KafkaConfluentSchemaRegistryJsonMessageDecoder implements StreamMessageDecoder<byte[]> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(KafkaConfluentSchemaRegistryJsonMessageDecoder.class);
+  private static final String SCHEMA_REGISTRY_REST_URL = "schema.registry.rest.url";
+  private static final String SCHEMA_REGISTRY_OPTS_PREFIX = "schema.registry.";
+  public static final String CACHED_SCHEMA_MAP_CAPACITY = "cached.schema.map.capacity";
+  public static final String DEFAULT_CACHED_SCHEMA_MAP_CAPACITY = "1000";
+  private KafkaJsonSchemaDeserializer _deserializer;
+  private RecordExtractor<Map<String, Object>> _jsonRecordExtractor;
+  private String _topicName;
+
+  public RestService createRestService(String schemaRegistryUrl, Map<String, String> configs) {
+    RestService restService = new RestService(schemaRegistryUrl);
+
+    ConfigDef configDef = new ConfigDef();
+    SslConfigs.addClientSslSupport(configDef);
+    Map<String, ConfigDef.ConfigKey> configKeyMap = configDef.configKeys();
+    Map<String, Object> sslConfigs = new HashMap<>();
+    for (String key : configs.keySet()) {
+      if (!key.equals(SCHEMA_REGISTRY_REST_URL) && key.startsWith(SCHEMA_REGISTRY_OPTS_PREFIX)) {
+        String value = configs.get(key);
+        String schemaRegistryOptKey = key.substring(SCHEMA_REGISTRY_OPTS_PREFIX.length());
+
+        if (configKeyMap.containsKey(schemaRegistryOptKey)) {
+          if (configKeyMap.get(schemaRegistryOptKey).type == ConfigDef.Type.PASSWORD) {
+            sslConfigs.put(schemaRegistryOptKey, new Password(value));
+          } else {
+            sslConfigs.put(schemaRegistryOptKey, value);
+          }
+        }
+      }
+    }
+
+    if (!sslConfigs.isEmpty()) {
+      DefaultSslEngineFactory sslFactory = new DefaultSslEngineFactory();
+      sslFactory.configure(sslConfigs);
+      restService.setSslSocketFactory(sslFactory.sslContext().getSocketFactory());
+    }
+    return restService;
+  }
+
+  @Override
+  public void init(Map<String, String> props, Set<String> fieldsToRead, String topicName)
+      throws Exception {
+    checkState(props.containsKey(SCHEMA_REGISTRY_REST_URL), "Missing required property '%s'", SCHEMA_REGISTRY_REST_URL);
+    String schemaRegistryUrl = props.get(SCHEMA_REGISTRY_REST_URL);
+    JsonSchemaProvider jsonSchemaProvider = new JsonSchemaProvider();
+    int identityMapCapacity = Integer.parseInt(
+            props.getOrDefault(CACHED_SCHEMA_MAP_CAPACITY, DEFAULT_CACHED_SCHEMA_MAP_CAPACITY));
+    SchemaRegistryClient schemaRegistryClient =
+        new CachedSchemaRegistryClient(createRestService(schemaRegistryUrl, props), identityMapCapacity,
+                Collections.singletonList(jsonSchemaProvider), props, null);
+
+    _deserializer = new KafkaJsonSchemaDeserializer(schemaRegistryClient);
+    Preconditions.checkNotNull(topicName, "Topic must be provided");
+    _topicName = topicName;
+    _jsonRecordExtractor = PluginManager.get().createInstance(JSONRecordExtractor.class.getName());
+    _jsonRecordExtractor.init(fieldsToRead, null);
+  }
+
+  @Override
+  public GenericRow decode(byte[] payload, GenericRow destination) {
+    try {
+      JsonNode jsonRecord = (JsonNode) _deserializer.deserialize(_topicName, payload);
+      Map<String, Object> from = JsonUtils.jsonNodeToMap(jsonRecord);
+      return _jsonRecordExtractor.extract(from, destination);
+    } catch (RuntimeException e) {
+      ignoreOrRethrowException(e);
+      return null;
+    } catch (Exception e) {
+      LOGGER.error("Caught exception while decoding row, discarding row. Payload is {}", new String(payload), e);
+      return null;
+    }
path: pinot-plugins/pinot-input-format/pinot-confluent-json/src/main/java/org/apache/pinot/plugin/inputformat/json/confluent/KafkaConfluentSchemaRegistryJsonMessageDecoder.java
user: login: r-reis
body: It was fixed in the new commit, but I had to add an IOException handle due to the jsonNodeToMap function.
created_at: 2025-03-18T22:35:17Z
author_association: CONTRIBUTOR

PR comments:
user: login: mayankshriv
created_at: 2025-03-14T14:43:29Z
body: Thanks for reporting the issue and proposing a fix.
author_association: CONTRIBUTOR
user: login: codecov-commenter
created_at: 2025-03-14T15:24:06Z
body: ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15273?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 63.70%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`31c7c28`)](https://app.codecov.io/gh/apache/pinot/commit/31c7c28c90fbf1db47f5d02800b154deaec10398?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1891 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master   #15273      +/-   ##
============================================
+ Coverage     61.75%   63.70%   +1.95%     
- Complexity      207     1461    +1254     
============================================
  Files          2436     2784     +348     
  Lines        133233   157017   +23784     
  Branches      20636    24086    +3450     
============================================
+ Hits          82274   100030   +17756     
- Misses        44911    49445    +4534     
- Partials       6048     7542    +1494     
```

| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+99.99%)` | :arrow_up: |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+99.99%)` | :arrow_up: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+99.99%)` | :arrow_up: |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.66% <ø> (+1.95%)` | :arrow_up: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.53% <ø> (+1.90%)` | :arrow_up: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.68% <ø> (+1.93%)` | :arrow_up: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.50% <ø> (+35.77%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.70% <ø> (+1.95%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.70% <ø> (+1.95%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.13% <ø> (+9.24%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15273/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.30% <ø> (+6.56%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15273?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: rajagopr
created_at: 2025-03-20T03:36:32Z
body: Please address the checkstyle failures.
author_association: CONTRIBUTOR