av9ash/GitBugs huggingface dataset:
Bug: GROUP_BY_POS_OUT_OF_RANGE if group by Literal
Status: Open | Priority: Major | Resolution: 
Created: 12/Sep/24 01:43
Affects Version/s: 3.5.3


{noformat}
spark-sql (default)> create table t1(id int);
spark-sql (default)> create table t2 as select id, -1 as x from t1 group by id, x;
[GROUP_BY_POS_OUT_OF_RANGE] GROUP BY position -1 is not in select list (valid range is [1, 2]).; line 1 pos 30
{noformat}
Bug: Resource leak of spark yarn allocator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/24 08:13
Affects Version/s: 3.5.0

When the target < running containers number, the assigned containers from the resource manager will be skipped, but these containers are not released by invoking the amClient.releaseAssignedContainer , that will make these containers reserved into the Yarn resourceManager at least 10 minutes. And so, the cluster resource will be wasted at a high ratio.

And this will reflect that the vcore * seconds statistics from yarn side will be greater than the result from the spark event logs.

From my statistics, the cluster resource waste ratio is ~25% if the spark jobs are exclusive in this cluster.

 

The more details could be found in this blog: [https://zuston.vercel.app/publish/resource-leak-of-spark-yarn-allocator/]
Bug: Increase PerMethodRecompilationCutoff and PerBytecodeRecompilationCutoff
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/24 00:50
Affects Version/s: 4.0.0

Cluster becomes slower and slower if do not disable {{spark.sql.codegen.wholeStage}}.

{noformat}
java -XX:+PrintFlagsFinal -version | grep Cutoff    
     intx LiveNodeCountInliningCutoff              = 40000                                  {C2 product} {default}
     intx PerBytecodeRecompilationCutoff           = 200                                       {product} {default}
     intx PerMethodRecompilationCutoff             = 400                                       {product} {default}
openjdk version "17.0.6" 2023-01-17 LTS
OpenJDK Runtime Environment Zulu17.40+20-SA (build 17.0.6+10-LTS)
OpenJDK 64-Bit Server VM Zulu17.40+20-SA (build 17.0.6+10-LTS, mixed mode, sharing)
{noformat}
Bug: Replace AnyRefMap with HashMap
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/24 09:50
Affects Version/s: 4.0.0

scala 2.13.8:

{noformat}
OpenJDK 64-Bit Server VM 1.8.0_382-b05 on Mac OS X 14.5
Apple M2 Max
Benchmark Map:                            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
HashMap                                            1121           1126           3          0.0      280264.2       1.0X
AnyRefMap                                          1590           1609          29          0.0      397386.4       0.7X
{noformat}

scala 2.12.18:
{noformat}
OpenJDK 64-Bit Server VM 1.8.0_382-b05 on Mac OS X 14.5
Apple M2 Max
Benchmark Map:                            Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
HashMap                                            1679           1689          11          0.0      419759.2       1.0X
AnyRefMap                                          1585           1592           8          0.0      396357.1       1.1X
{noformat}

More details:
https://github.com/scala/bug/issues/13029#issuecomment-2325489902
Bug: Introducing PySpark Plotting API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/24 06:10
Affects Version/s: 4.0.0

Pandas on Spark DataFrame supports plotting, but PySpark DataFrame does not. Enabling plotting for PySpark DataFrame can enhance data visualization capabilities, making it easier for users to explore large datasets and gain insights directly from PySpark DataFrames.
Bug: Remove the ANSI config suggestion in CANNOT_PARSE_TIMESTAMP
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 10:11
Affects Version/s: 4.0.0

CANNOT_PARSE_TIMESTAMP contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: Zookeeper node interruption causes Active spark master to exit
Status: Open | Priority: Major | Resolution: 
Created: 09/Sep/24 11:46
Affects Version/s: 3.4.2

Shutting Down a single zookeeper node caused spark master to exit.  The master should have connected to a second zookeeper node. 

{code:title=log output}
16/05/25 18:21:28 INFO master.Master: Launching executor app-20160525182128-0006/1 on worker worker-20160524013212-10.16.28.76-59138
16/05/25 18:21:28 INFO master.Master: Launching executor app-20160525182128-0006/2 on worker worker-20160524013204-10.16.21.217-47129
16/05/26 00:16:01 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x154dfc0426b0054, likely server has closed socket, closing socket connection and attempting reconnect
16/05/26 00:16:01 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x254c701f28d0053, likely server has closed socket, closing socket connection and attempting reconnect
16/05/26 00:16:01 INFO state.ConnectionStateManager: State change: SUSPENDED
16/05/26 00:16:01 INFO state.ConnectionStateManager: State change: SUSPENDED
16/05/26 00:16:01 INFO master.ZooKeeperLeaderElectionAgent: We have lost leadership
16/05/26 00:16:01 ERROR master.Master: Leadership has been revoked -- master shutting down. }}
{code}

spark-env.sh: 

{code:title=spark-env.sh}
export SPARK_LOCAL_DIRS=/ephemeral/spark/local
export SPARK_WORKER_DIR=/ephemeral/spark/work
export SPARK_LOG_DIR=/var/log/spark
export HADOOP_CONF_DIR=/home/ubuntu/hadoop-2.6.3/etc/hadoop

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=gn5456-zookeeper-01:2181,gn5456-zookeeper-02:2181,gn5456-zookeeper-03:2181"

export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true"
{code}
Bug: Remove the ANSI config suggestion in INVALID_ARRAY_INDEX and INVALID_ARRAY_INDEX_IN_ELEMENT_AT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 10:28
Affects Version/s: 4.0.0

INVALID_ARRAY_INDEX and INVALID_ARRAY_INDEX_IN_ELEMENT_AT contain suggested fixes for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: SPJ: Skip shuffles if possible for sorts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/24 20:43
Affects Version/s: 4.0.0

It may be possible to use the same logic as SPJ (for joins), to skip shuffles for ORDER BY or other sort operations, if we are sorting on partition columns.

Today all data is shuffled even if we have all the partition values and can return them in sorted order.
Bug: JsonProtocol should write 'Shuffle Push Read Metrics' and 'Merged Fallback Count' field of shuffle read metrics
Status: Open | Priority: Major | Resolution: 
Created: 13/Sep/24 03:44
Affects Version/s: 4.0.0

`JsonProtocol` writes 'Push Based Shuffle' and 'Merged Fetch Fallback Count' field of shuffle read metrics at present, which is inconsistent with fields of task metric from json. Therefore,  `JsonProtocol` should write 'Shuffle Push Read Metrics' and 'Merged Fallback Count' field of shuffle read metrics.
Bug: UDT type is not expanded into its StructType in the schema definition
Status: Open | Priority: Major | Resolution: 
Created: 18/Sep/24 01:57
Affects Version/s: 4.0.0

A UDT type field does not show up as constituent struct type in the schema. Instead it shows up as UserDefinedType class .

For eg: it shows up as

root
 |-- message: test (nullable = true)

 

But mesage is a field of type UDT with schema represented as 

StructField("intField", IntegerType, nullable = false),
StructField("stringField", StringType, nullable = false)))

so message field should be a struct type, with schema as

root
 |-- message: struct (nullable = true)
 |    |-- intField: integer (nullable = false)
 |    |-- stringField: string (nullable = false)

 

will be opening a PR and along with bug test
Bug: Use agnostic encoders instead of expression encoders in user facing API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/24 00:27
Affects Version/s: 4.0.0

ExpressionEncoders are quite heavily used in the classic Dataset and KeyValueGroupedDataset. Because of the move to sql/api most of the user provided are in fact AgnosticEncoders, this is mostly fine but in some cases this leads to situations where we try to combine the two which does not work.

Actual ExpressionEncoders are generally only needed inside the engine, or for collecting results. We should try to limit them to these use cases.
Bug: Add spark.cleaner.ttl functionality for long lived jobs
Status: Open | Priority: Major | Resolution: 
Created: 25/Sep/24 19:41
Affects Version/s: 4.0.0

We should add a TTL (and maybe a threshold?) to clean shuffle files which have been stored for greater than some fixed period of time. This would be useful for long lived jobs with Spark Connect and Notebooks where items may not go out-of-scope and be resident for longer than needed.

 

See SPARK-7689 which removed the original TTL cleaner.

 

To reduce the chance of confusion/difficulty to understand which was part of the original reason for removing the TTL based cleaner we should (instead of "raw" TTL) keep track of when it was accessed last to reset the counter + log on removal.
Bug: Cast between UDT and other types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/24 19:41
Affects Version/s: 4.0.0

Our customer faced an error when migrating queries that write UDT column from Hive to Iceberg table.

The error happens when Spark tries to cast UDT column to the data type (i.e., the sql type of the UDT) of the table column. The cast is added by table column resolution rule for V2 writing commands.

Currently `Cast` expression doesn't support casting between UDT and other types.
Bug: Feature transformers are case sensitive when unintented
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/24 12:30
Affects Version/s: 3.4.3

Hi team,

https://spark.apache.org/docs/latest/ml-features

The feature transformers are case sensitive even though the configuration 

 
{code:java}
spark.conf.get("spark.sql.caseSensitive") {code}
 

is set to false. The user of all these transformers are forced to abide by case of the column in the dataframe

 
{code:java}
 val data = List(Row("the movie was great", "positive", 10, "greatest of all time"),
    Row("the movie was average", "negative", 11, "just average things, average storyline"),
    Row("movie was fun", "positive", 2, "superb screen play"))
  val schema = new StructType()
    .add("comments", StringType, true)
    .add("reviews", StringType, true)
    .add("counts", IntegerType, true)
    .add("Additional_COMMENTS", StringType, true)
val df = spark.createDataFrame(data.asJava, schema)
  val si = new StringIndexer().setInputCol("additional_comments").setOutputCol("si_additional_comments")
  si.fit(df).transform(df).show() {code}
The above code fails with 
{code:java}
Exception in thread "main" org.apache.spark.SparkException: Input column additional_comments does not exist.
    at org.apache.spark.ml.feature.StringIndexerBase.$anonfun$validateAndTransformSchema$2(StringIndexer.scala:128)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema(StringIndexer.scala:123)
    at org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema$(StringIndexer.scala:115) {code}
Which means that the column "additional_comments" needs to be provided in the same case as in the dataframe. 

 

I think when the caseSensitive  setting is set to false we should be able to use the naming in any case.

 


Can someone please help to solve this bug for all transformers.?
Bug: Remove the ANSI config suggestion in INVALID_FRACTION_OF_SECOND
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 10:31
Affects Version/s: 4.0.0

INVALID_FRACTION_OF_SECOND contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: Support pie subplots with plotly backend
Status: Open | Priority: Major | Resolution: 
Created: 26/Sep/24 07:03
Affects Version/s: 4.0.0

In Pandas
{code:java}
>>> import pandas as pd
>>> pd.options.plotting.backend = "plotly"
>>> df = pd.DataFrame({'mass': [0.330, 4.87 , 5.97],
...                    'radius': [2439.7, 6051.8, 6378.1]},
...                   index=['Mercury', 'Venus', 'Earth'])
>>> df
         mass  radius
Mercury  0.33  2439.7
Venus    4.87  6051.8
Earth    5.97  6378.1
>>> fig = df.plot.pie()
Traceback (most recent call last):
...
    raise ValueError("pie requires either y column or 'subplots=True'")
ValueError: pie requires either y column or 'subplots=True'
>>> fig = df.plot.pie(subplots=True)
Traceback (most recent call last):
...
NotImplementedError: kind='pie' not yet supported for plotting.backend='plotly'
>>> pd.__version__
'2.2.2' {code}
Bug: Remove the ANSI config suggestion in INVALID_URL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 10:33
Affects Version/s: 4.0.0

INVALID_URL contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try_*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: ArrayRemove() Function Need Remove NULL Value
Status: Open | Priority: Major | Resolution: 
Created: 05/Sep/24 03:33
Affects Version/s: 3.2.1

I want to calculate the intersection of two arrays like this: 
{noformat}
select   
    case when intersect_size > 0 then 1 else 0 end as is_include
from ( 
    select 
        size(array_intersect(array_a, array_b)) as intersect_size 
from table_a
)
{noformat}
 
But, the NULL will affect the output:
{code:java}
SELECT size(array_intersect(array(1, 2, 3, null), array(null)))

Output: 1 {code}
So I want remove the NULL in first array by using {*}array_remove{*}：
{code:java}
SELECT array_remove(array(1, 2, 3, null, 3), null) 

Output: null{code}
I want to add extra logic for function *array_remove* to remove NULL. Shall I overwrite the function (May be named: array_remove(array_a, array_b, isIgnoreNull)) or just fix the original function?
Bug: Optimized hadoop-aws dependency, aws-java-sdk-bundle jar is too large
Status: Open | Priority: Major | Resolution: 
Created: 04/Sep/24 04:42
Affects Version/s: 3.5.2

aws-java-sdk-bundle jar is too large，The size of the spark image will double。hadoop aws only requires the use of aws-java-sdk-s3 and aws-java-sdk-dynamodb

 
{code:java}
// code placeholder
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-aws</artifactId>
    <version>${hadoop.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bundle</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>${awssdk.v1.version}</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-dynamodb</artifactId>
    <version>${awssdk.v1.version}</version>
</dependency> {code}
Bug: Merge _LEGACY_ERROR_TEMP_2042 into ARITHMETIC_OVERFLOW
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/24 11:49
Affects Version/s: 4.0.0

_LEGACY_ERROR_TEMP_2042 contains suggested fix for turning off ANSI mode. Now that in Spark 4.0.0 we have moved to ANSI mode on by default, we want to keep suggestions of this kind to the minimum. There exist implementations of `try*` functions which provide safe way to get behaviour as for ANSI mode off and suggestions of this kind should be sufficient.
Bug: EventTimeWatermark to support TimestampNTZType
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/24 19:58
Affects Version/s: 3.4.0

Spark added TimestampNTZType support since 3.4.0. [https://spark.apache.org/docs/3.4.0/api/java/org/apache/spark/sql/types/TimestampNTZType.html]

However, EventTimeWatermark with TimestampNTZType fails analysis check.

 
{code:java}
case etw: EventTimeWatermark =>
            etw.eventTime.dataType match {
              case s: StructType
                if s.find(_.name == "end").map(_.dataType) == Some(TimestampType) =>
              case _: TimestampType =>
              case _ =>
                etw.failAnalysis(
                  errorClass = "EVENT_TIME_IS_NOT_ON_TIMESTAMP_TYPE",
                  messageParameters = Map(
                    "eventName" -> toSQLId(etw.eventTime.name),
                    "eventType" -> toSQLType(etw.eventTime.dataType)))
            } {code}
Both *FilterPushdownSuite.scala* and *AnalysisSuite.scala* don't explicitly avoid timestampNTZ. Moreover, from *StatefulAggregationStrategy* --> {*}EventTimeWatermarkExec{*}, even if the eventTime is of type TimestampNTZType, it should be handled as per {*}datetimeExpressions{*}.{*}scala{*}. It is also supported in Window Expressions {*}WindowSpecDefinition ->  isValidFrameType{*}.
Bug: Enable PredictBatchUDFTests.test_caching for NumPy 2
Status: Open | Priority: Major | Resolution: 
Created: 26/Sep/24 02:50
Affects Version/s: 4.0.0

 
{code:java}
import numpy as np
import pandas as pd
from pyspark.ml.functions import predict_batch_udf
from pyspark.sql.types import DoubleType
from pyspark.sql.functions import struct
data = np.arange(0, 36, dtype=np.float64).reshape(-1, 4)
pdf = pd.DataFrame(data, columns=["a", "b", "c", "d"])
df = spark.createDataFrame(pdf)
def make_predict_fn():
    fake_output = np.random.random()
    def predict(inputs):
        return np.array([fake_output for i in inputs])
    return predict
 
identity = predict_batch_udf(make_predict_fn, return_type=DoubleType(), batch_size=5)
df1 = df.withColumn("preds", identity(struct("a"))).toPandas()
df2 = df.withColumn("preds", identity(struct("a"))).toPandas()
{code}
NumPy 2.1.0
{code:java}
>>> df1
      a     b     c     d     preds
0   0.0   1.0   2.0   3.0  0.431752
1   4.0   5.0   6.0   7.0  0.912097
2   8.0   9.0  10.0  11.0  0.679628
3  12.0  13.0  14.0  15.0  0.853850
4  16.0  17.0  18.0  19.0  0.389971
5  20.0  21.0  22.0  23.0  0.654521
6  24.0  25.0  26.0  27.0  0.430569
7  28.0  29.0  30.0  31.0  0.331055
8  32.0  33.0  34.0  35.0  0.306073
>>> df2
      a     b     c     d     preds
0   0.0   1.0   2.0   3.0  0.679628
1   4.0   5.0   6.0   7.0  0.430569
2   8.0   9.0  10.0  11.0  0.853850
3  12.0  13.0  14.0  15.0  0.306073
4  16.0  17.0  18.0  19.0  0.654521
5  20.0  21.0  22.0  23.0  0.389971
6  24.0  25.0  26.0  27.0  0.507598
7  28.0  29.0  30.0  31.0  0.912097
8  32.0  33.0  34.0  35.0  0.431752 {code}
which should be
{code:java}
>>> df1
      a     b     c     d     preds
0   0.0   1.0   2.0   3.0  0.685941
1   4.0   5.0   6.0   7.0  0.685941
2   8.0   9.0  10.0  11.0  0.685941
3  12.0  13.0  14.0  15.0  0.685941
4  16.0  17.0  18.0  19.0  0.685941
5  20.0  21.0  22.0  23.0  0.685941
6  24.0  25.0  26.0  27.0  0.685941
7  28.0  29.0  30.0  31.0  0.685941
8  32.0  33.0  34.0  35.0  0.685941
>>> df2
      a     b     c     d     preds
0   0.0   1.0   2.0   3.0  0.685941
1   4.0   5.0   6.0   7.0  0.685941
2   8.0   9.0  10.0  11.0  0.685941
3  12.0  13.0  14.0  15.0  0.685941
4  16.0  17.0  18.0  19.0  0.685941
5  20.0  21.0  22.0  23.0  0.685941
6  24.0  25.0  26.0  27.0  0.685941
7  28.0  29.0  30.0  31.0  0.685941
8  32.0  33.0  34.0  35.0  0.685941 {code}
Bug: Deprecate getErrorClass in SparkThrowable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/24 09:54
Affects Version/s: 4.0.0

As a follow up of the conclusion of https://issues.apache.org/jira/browse/SPARK-46810:
# Deprecate the getErrorClass() method in SparkThrowable
# Add new method getCondition as the replacement of getErrorClass
Bug: Introduce Shim for missing spark/core classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Sep/24 00:15
Affects Version/s: 4.0.0

Introduce shims for SparkContext, RDD, and QueryExecution. This will have to be a in a separate module, and is supposed to be a compile time dependency for sql/api project, and an actual dependency for an independent Spark Connect Client.

We need these three classes to support all user facing API in the sql/api project. This will allow us to make the classes the primary interface for Scala Dataset operations.

For connect these methods will throw an (actionable) error, and for the classic client they will just work. On the connect side in the future we can use this to build better errors, and provide (method specific) mitigations.
Bug: Cannot generate code for expression: years
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/24 07:16
Affects Version/s: 4.0.0

The example below portraits the issue:

{code:scala}
scala> spark.range(10).select(partitioning.years($"id")).show()
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for expression: years(input[0, bigint, false]) SQLSTATE: XX000
  at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
  at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
{code}
Bug: uncaught Java exception from make_timestamp() with bad timezone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Sep/24 21:42
Affects Version/s: 4.0.0

spark-sql (default)> select make_timestamp(1, 2, 28, 23, 1, 1, -100);
24/09/23 19:35:44 ERROR SparkSQLDriver: Failed in [select make_timestamp(1, 2, 28, 23, 1, 1, -100)]
java.time.DateTimeException: Invalid ID for ZoneOffset, invalid format: -100
at java.base/java.time.ZoneOffset.of(ZoneOffset.java:243)
at java.base/java.time.ZoneId.of(ZoneId.java:404)
Bug: A Bean class with a serializable POJO field looses data when converted back from dataframe,  as dataset 
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/24 04:00
Affects Version/s: 3.5.2

If a Bean class contains a serializable POJO  ( without any getter/setter) property,  the encoder does not convert it into Binary Type, and looses the information.

Using that encoder , we can convert the Dataset of Bean to dataframe.

But When the dataframe is converted back to dataset, the serializable Pojo field  in the bean remains null. 

The issue is addressed along with the PR for issue https://issues.apache.org/jira/browse/SPARK-49789

Will be adding the bug test in the PR