av9ash/GitBugs huggingface dataset:
Bug: Support data type Timestamp(NANOSECOND, null)
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 03/Jan/23 02:18
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 1966, in pyspark.sql.connect.functions.hour
Failed example:
    df.select(hour('ts').alias('hour')).collect()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.hour[2]>", line 1, in <module>
        df.select(hour('ts').alias('hour')).collect()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1017, in collect
        pdf = self.toPandas()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1031, in toPandas
        return self._session.client.to_pandas(query)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 623, in _handle_error
        raise SparkConnectException(status.message, info.reason) from None
    pyspark.sql.connect.client.SparkConnectException: (org.apache.spark.SparkUnsupportedOperationException) Unsupported data type: Timestamp(NANOSECOND, null){code}
Bug: df["bad_key"] does not raise AnalysisException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 05:41
Affects Version/s: 3.4.0

e.g.)

{code}
23/01/12 14:33:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
FAILED [  8%]
pyspark/sql/tests/test_column.py:105 (ColumnParityTests.test_access_column)
self = <pyspark.sql.tests.connect.test_parity_column.ColumnParityTests testMethod=test_access_column>

    def test_access_column(self):
        df = self.df
        self.assertTrue(isinstance(df.key, Column))
        self.assertTrue(isinstance(df["key"], Column))
        self.assertTrue(isinstance(df[0], Column))
        self.assertRaises(IndexError, lambda: df[2])
>       self.assertRaises(AnalysisException, lambda: df["bad_key"])
E       AssertionError: AnalysisException not raised by <lambda>

../test_column.py:112: AssertionError
{code}
Bug: Add a config flag to force exit on JDK major version mismatch
Status: Open | Priority: Major | Resolution: 
Created: 12/Jan/23 18:00
Affects Version/s: 3.4.0

JRE version mismatches can cause errors which are difficult to debug (potentially correctness with serialization issues). We should add a flag for platform which wish to "fail fast" and exit on major version mismatch.

 

I think this could be a good thing to have default on in Spark 4.

 

Generally I expect to see more folks upgrading JRE & JDKs in the coming few years.
Bug: Wrong result when parsing a multiline JSON file with differing types for same column
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/23 21:21
Affects Version/s: 3.2.1

Here is a simple reproduction of the problem. We have a JSON file whose content looks like following and is in multiLine format.
{code}
[{"name":""},{"name":123.34}]
{code}

Here is the result of spark query when we read the above content.

scala> val df = spark.read.format("json").option("multiLine", true).load("/tmp/json")
df: org.apache.spark.sql.DataFrame = [name: double]

scala> df.show(false)
+----+
|name|
+----+
|null|
+----+


scala> df.count
res5: Long = 2

This is quite a serious problem for us as it's causing us to master corrupt data in lake. If there is some issue with parsing the input, we expect spark set the "_corrupt_record" so that we can act on it. Please note that df.count is reporting 2 rows where as df.show only reports 1 row with null value.
Bug: CreateDataframe from Pandas with Struct and Timestamp
Status: Open | Priority: Major | Resolution: 
Created: 12/Jan/23 09:39
Affects Version/s: 3.4.0

The following should be supported and correctly truncate the nanosecond timestamps.

{code:python}
from datetime import datetime, timezone, timedelta
from pandas import Timestamp

ts=Timestamp(year=2019, month=1, day=1, nanosecond=500, tz=timezone(timedelta(hours=-8)))

d = pd.DataFrame({"col1": [1], "col2": [{"a":1, "b":2.32, "c":ts}]})
spark.createDataFrame(d).collect()

{code}
Bug: Test parity: pyspark.sql.tests.test_group, test_serde, test_datasources and test_column
Status: Resolved | Priority: Major | Resolution: Done
Created: 12/Jan/23 04:57
Affects Version/s: 3.4.0

See https://issues.apache.org/jira/browse/SPARK-41652 and https://issues.apache.org/jira/browse/SPARK-41651
Bug: Throw proper errors in Dataset.to()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 03:41
Affects Version/s: 3.4.0

{code:java}
schema = StructType(
    [StructField("i", StringType(), True), StructField("j", IntegerType(), True)]
)
df = self.spark.createDataFrame([("a", 1)], schema)

schema1 = StructType([StructField("j", StringType()), StructField("i", StringType())])
df1 = df.to(schema1)
self.assertEqual(schema1, df1.schema)
self.assertEqual(df.count(), df1.count())

schema2 = StructType([StructField("j", LongType())])
df2 = df.to(schema2)
self.assertEqual(schema2, df2.schema)
self.assertEqual(df.count(), df2.count())

schema3 = StructType([StructField("struct", schema1, False)])
df3 = df.select(struct("i", "j").alias("struct")).to(schema3)
self.assertEqual(schema3, df3.schema)
self.assertEqual(df.count(), df3.count())

# incompatible field nullability
schema4 = StructType([StructField("j", LongType(), False)])
self.assertRaisesRegex(
    AnalysisException, "NULLABLE_COLUMN_OR_FIELD", lambda: df.to(schema4)
){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 1486, in test_to
    self.assertRaisesRegex(
AssertionError: AnalysisException not raised by <lambda> {code}
Bug: Wrap InMemoryTableScanExec with QueryStage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/23 10:09
Affects Version/s: 3.5.0

The first access to the cached plan which is enable AQE is tricky. Currently, we can not preserve its output partitioning and ordering.

The whole query plan also missed lots of optimization in AQE framework. Wrap 
InMemoryTableScanExec  to query stage can resolve all these issues.
Bug: Upgrade kubernetes-client from 6.3.1 to 6.4.0
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Jan/23 19:28
Affects Version/s: 3.4.0

A new version of kubernetes-client
According to SNYK this release fix a problem with [com.squareup.okhttp3:okhttp|https://security.snyk.io/vuln/SNYK-JAVA-COMSQUAREUPOKHTTP3-2958044] 

[Release notes|https://github.com/fabric8io/kubernetes-client/releases/tag/v6.4.0 ]
Bug: Spark 3.3.0, Error with java.io.IOException: Mkdirs failed to create file
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 20/Jan/23 04:16
Affects Version/s: 3.3.0

23/01/18 20:23:24 WARN TaskSetManager: Lost task 4.0 in stage 0.0 (TID 4) (10.64.109.72 executor 0): java.io.IOException: Mkdirs failed to create file:/var/backup/_temporary/0/_temporary/attempt_202301182023173234741341853025716_0005_m_000004_0 (exists=false, cwd=file:/opt/spark-3.3.0/work/app-20230118202317-0001/0)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:515)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1081)
        at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:113)
        at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:238)
        at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:126)
        at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Bug: Implement DataFrame `toLocalIterator`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 05:10
Affects Version/s: 3.4.0

{code:java}
schema = StructType(
    [StructField("i", StringType(), True), StructField("j", IntegerType(), True)]
)
df = self.spark.createDataFrame([("a", 1)], schema)

schema1 = StructType([StructField("j", StringType()), StructField("i", StringType())])
df1 = df.to(schema1)
self.assertEqual(schema1, df1.schema)
self.assertEqual(df.count(), df1.count())

schema2 = StructType([StructField("j", LongType())])
df2 = df.to(schema2)
self.assertEqual(schema2, df2.schema)
self.assertEqual(df.count(), df2.count())

schema3 = StructType([StructField("struct", schema1, False)])
df3 = df.select(struct("i", "j").alias("struct")).to(schema3)
self.assertEqual(schema3, df3.schema)
self.assertEqual(df.count(), df3.count())

# incompatible field nullability
schema4 = StructType([StructField("j", LongType(), False)])
self.assertRaisesRegex(
    AnalysisException, "NULLABLE_COLUMN_OR_FIELD", lambda: df.to(schema4)
){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 1486, in test_to
    self.assertRaisesRegex(
AssertionError: AnalysisException not raised by <lambda> {code}
Bug: Allow `columns` parameter when creating DataFrame with Series.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/23 02:32
Affects Version/s: 3.5.0

pandas API on Spark doesn't allow creating DataFrame with Series by specifying the `columns` parameter as below:
{code:java}
>>> ps.DataFrame(psser, columns=["labels"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../spark/python/pyspark/pandas/frame.py", line 539, in __init__
    assert columns is None
AssertionError {code}
We should make it available.
Bug: createDataFrame with UDT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 05:52
Affects Version/s: 3.4.0

{code}

pyspark/sql/tests/test_types.py:596 (TypesParityTests.test_apply_schema_with_udt)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_apply_schema_with_udt>

    def test_apply_schema_with_udt(self):
        row = (1.0, ExamplePoint(1.0, 2.0))
        schema = StructType(
            [
                StructField("label", DoubleType(), False),
                StructField("point", ExamplePointUDT(), False),
            ]
        )
>       df = self.spark.createDataFrame([row], schema)

../test_types.py:605: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:282: in createDataFrame
    _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert ExamplePoint(1.0,2.0) with type ExamplePoint: did not recognize Python value type when inferring an Arrow data type

pyarrow/error.pxi:100: ArrowInvalid

{code}
Bug: Support DataFrameWriter.saveAsTable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 15:43
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/readwriter.py", line 369, in pyspark.sql.connect.readwriter.DataFrameWriter.insertInto
Failed example:
    df.write.saveAsTable("tblA")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.readwriter.DataFrameWriter.insertInto[2]>", line 1, in <module>
        df.write.saveAsTable("tblA")
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/readwriter.py", line 350, in saveAsTable
        self._spark.client.execute_command(self._write.command(self._spark.client))
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 459, in execute_command
        self._execute(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 547, in _execute
        self._handle_error(rpc_error)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 623, in _handle_error
        raise SparkConnectException(status.message, info.reason) from None
    pyspark.sql.connect.client.SparkConnectException: (java.lang.ClassNotFoundException) .DefaultSource{code}
Bug: Implement SparkSession.udf
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 02:22
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 2331, in pyspark.sql.connect.functions.call_udf
Failed example:
    _ = spark.udf.register("intX2", lambda i: i * 2, IntegerType())
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.call_udf[3]>", line 1, in <module>
        _ = spark.udf.register("intX2", lambda i: i * 2, IntegerType())
    AttributeError: 'SparkSession' object has no attribute 'udf'{code}
Bug: Kryo ClassCastException getting task result when JDK versions mismatch
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 12/Jan/23 19:29
Affects Version/s: 3.3.0

{noformat}
22/12/21 01:27:12 ERROR TaskResultGetter: Exception while getting task result
com.esotericsoftware.kryo.KryoException: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.nio.ByteBuffer
Serialization trace:
lowerBounds (org.apache.iceberg.GenericDataFile)
taskFiles (org.apache.iceberg.spark.source.SparkWrite$TaskCommit)
writerCommitMessage (org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
{noformat}
Iceberg 1.1 `BaseFile.lowerBounds` is defined as
{code:java}
Map<Integer, ByteBuffer> {code}
Driver JDK version: 1.8.0_352 (Azul Systems, Inc.)
Executor JDK version: openjdk version "17.0.5" 2022-10-18 LTS

Kryo version: 4.0.2

 

Same Spark job works when both driver and executors run the same JDK 8 or JDK 17.
Bug: spark.read fails to read filenames with accented characters
Status: Open | Priority: Major | Resolution: 
Created: 26/Jan/23 08:16
Affects Version/s: 3.2.1

Unable to read filenames with accented characters in the filename.

*Sample error:*
{code:java}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 43 in stage 1.0 failed 4 times, most recent failure: Lost task 43.3 in stage 1.0 (TID 105) (10.139.64.5 executor 0): java.io.FileNotFoundException: /4842022074360943/user/hive/warehouse/hls_cms_source.db/raw_files/synthea_mass/ccda/Amalia471_Magaña874_3912696a-0aef-492e-83ef-468262b82966.xml{code}
 

*{{Steps to reproduce error:}}*
{code:java}
%sh
mkdir -p /dbfs/user/hive/warehouse/hls_cms_source.db/raw_files/synthea_mass
wget  https://synthetichealth.github.io/synthea-sample-data/downloads/synthea_sample_data_ccda_sep2019.zip -O ./synthea_sample_data_ccda_sep2019.zip 
unzip ./synthea_sample_data_ccda_sep2019.zip -d /dbfs/user/hive/warehouse/hls_cms_source.db/raw_files/synthea_mass/
{code}
 
{code:java}
spark.conf.set("spark.sql.caseSensitive", "true")
df = (
  spark.read.format('xml')
   .option("rowTag", "ClinicalDocument")
  .load('/user/hive/warehouse/hls_cms_source.db/raw_files/synthea_mass/ccda/')
){code}
Is there a way to deal with this situation where I don't have control over the file names for some reason?
Bug: createDataFrame should autogenerate missing column names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 06:03
Affects Version/s: 3.4.0

{code}
pyspark/sql/tests/test_types.py:233 (TypesParityTests.test_infer_schema_not_enough_names)
['col1', '_2'] != ['col1']

Expected :['col1']
Actual   :['col1', '_2']
<Click to see difference>

self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_not_enough_names>

    def test_infer_schema_not_enough_names(self):
        df = self.spark.createDataFrame([["a", "b"]], ["col1"])
>       self.assertEqual(df.columns, ["col1", "_2"])

../test_types.py:236: AssertionError

{code}
Bug: Better Scala Client Integration test
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Jan/23 16:32
Affects Version/s: 3.4.0

The current Scala client has a few integration tests that requires a build first before running client tests. This is not very nice to maven developers as they will not be able to do a `mvn clean install` to run all tests.

 

Look into marking these test as ITs and other better ways for maven to run test after packages are built.

 

Make sure the test run in SBT as well.
Bug: DataFrame.join creating ambiguous column names
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 02/Jan/23 17:05
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 254, in pyspark.sql.connect.dataframe.DataFrame.drop
Failed example:
    df.join(df2, df.name == df2.name, 'inner').drop('name').show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.drop[5]>", line 1, in <module>
        df.join(df2, df.name == df2.name, 'inner').drop('name').show()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 534, in show
        print(self._show_string(n, truncate, vertical))
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 423, in _show_string
        ).toPandas()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1031, in toPandas
        return self._session.client.to_pandas(query)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 619, in _handle_error
        raise SparkConnectAnalysisException(
    pyspark.sql.connect.client.SparkConnectAnalysisException: [AMBIGUOUS_REFERENCE] Reference `name` is ambiguous, could be: [`name`, `name`].
    Plan: {code}
Bug: Handle duplicate columns in `createDataFrame`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 02:56
Affects Version/s: 3.4.0

{code:java}
df = self.spark.createDataFrame([(1, 2)], ["c", "c"]){code}
Error:
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 65, in test_duplicated_column_names
    df = self.spark.createDataFrame([(1, 2)], ["c", "c"])
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", line 277, in createDataFrame
    raise ValueError(
ValueError: Length mismatch: Expected axis has 1 elements, new values have 2 elements{code}
Bug: Support data type Duration(NANOSECOND)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 02:37
Affects Version/s: 3.4.0

{code:java}
import pandas as pd
from datetime import timedelta

df = self.spark.createDataFrame(pd.DataFrame({"a": [timedelta(microseconds=123)]})) {code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 1291, in test_create_dataframe_from_pandas_with_day_time_interval
    self.assertEqual(df.toPandas().a.iloc[0], timedelta(microseconds=123))
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1031, in toPandas
    return self._session.client.to_pandas(query)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 413, in to_pandas
    return self._execute_and_fetch(req)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 623, in _handle_error
    raise SparkConnectException(status.message, info.reason) from None
pyspark.sql.connect.client.SparkConnectException: (org.apache.spark.SparkUnsupportedOperationException) Unsupported data type: Duration(NANOSECOND){code}
Bug: Add built-in table-valued functions posexplode and posexplode_outer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/23 22:03
Affects Version/s: 3.4.0

Add `posexplode` and `posexplode_outer` to the built-in table function registry.

Add new SQL tests in `table-valued-functions.sql` and `join-lateral.sql`.
Bug: Implement SparkSession.conf
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 18:47
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 2119, in pyspark.sql.connect.functions.unix_timestamp
Failed example:
    spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.unix_timestamp[0]>", line 1, in <module>
        spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
    AttributeError: 'SparkSession' object has no attribute 'conf'{code}