Bug: pyspark sql
Status: Open | Priority: Minor | Resolution: 
Created: 17/Jun/23 13:49
Affects Version/s: 3.4.0

h3. Background

I am new to Spark and I am trying to use PySpark to read data from a Kafka topic but got some errors that can't search to get the answer. 
h3. How to reproduce?
 # Follow the [https://kafka.apache.org/quickstart] to start the kafka broker server.
 # Follow the [https://spark.apache.org/docs/latest/index.html#running-the-examples-and-shell] to start the PySpark shell
 # Follow the [https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html] to write the pyspark code and use spark-submit to submit the python code. 

My python code
{code:java}
from pyspark.sql import SparkSession 

spark = SparkSession.Builder().appName('Kafka').master('local').config("spark.driver.bindAddress","127.0.0.1").getOrCreate()

df = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "127.0.0.1:9092").option("subscribe", "quickstart-events").option("startingOffsets", "learliest").load()

df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")

df.show(){code}
How I submit the job
{code:java}
bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.13:3.4.0 kafka.py{code}
And the error I got 
{code:java}
23/06/17 21:47:16 ERROR Inbox: Ignoring error
java.lang.NullPointerException: Cannot invoke "org.apache.spark.storage.BlockManagerId.executorId()" because "idWithoutTopologyInfo" is null
    at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:600)
    at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:123)
    at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
    at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
    at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1623)
23/06/17 21:47:16 WARN Executor: Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Exception thrown in awaitResult:
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:101)
    at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:85)
    at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:80)
    at org.apache.spark.storage.BlockManager.reregister(BlockManager.scala:641)
    at org.apache.spark.executor.Executor.reportHeartBeat(Executor.scala:1111)
    at org.apache.spark.executor.Executor.$anonfun$heartbeater$1(Executor.scala:244)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2088)
    at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.spark.storage.BlockManagerId.executorId()" because "idWithoutTopologyInfo" is null
    at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:600)
    at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:123)
    at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
    at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
    at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
    ... 3 more
^CERROR:root:KeyboardInterrupt while sending command.
Traceback (most recent call last):
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 511, in send_command
    answer = smart_decode(self.stream.readline()[:-1])
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Traceback (most recent call last):
  File "/Users/ericsyh/code/PySpark-Learning/kafka.py", line 7, in <module>
    getOrCreate()
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 477, in getOrCreate
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line 512, in getOrCreate
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line 200, in __init__
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line 287, in _do_init
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/context.py", line 417, in _initialize_context
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1586, in __call__
23/06/17 21:47:21 INFO DiskBlockManager: Shutdown hook called
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
  File "/Users/ericsyh/code/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 511, in send_command
  File "/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^{code}