av9ash/GitBugs huggingface dataset:
Bug: DataFrame `toPandas` parity in return types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 15:03
Affects Version/s: 3.4.0

{code:java}
import numpy as np
import pandas as pd

df = self.spark.createDataFrame(
    [[[("a", 2, 3.0), ("a", 2, 3.0)]], [[("b", 5, 6.0), ("b", 5, 6.0)]]],
    "array_struct_col Array<struct<col1:string, col2:long, col3:double>>",
)
for is_arrow_enabled in [True, False]:
    with self.sql_conf({"spark.sql.execution.arrow.pyspark.enabled": is_arrow_enabled}):
        pdf = df.toPandas()
        self.assertEqual(type(pdf), pd.DataFrame)
        self.assertEqual(type(pdf["array_struct_col"]), pd.Series)
        if is_arrow_enabled:
            self.assertEqual(type(pdf["array_struct_col"][0]), np.ndarray)
        else:
            self.assertEqual(type(pdf["array_struct_col"][0]), list){code}
{code:java}
Traceback (most recent call last):
1415  File "/__w/spark/spark/python/pyspark/sql/tests/test_dataframe.py", line 1202, in test_to_pandas_for_array_of_struct
1416    df = self.spark.createDataFrame(
1417  File "/__w/spark/spark/python/pyspark/sql/connect/session.py", line 264, in createDataFrame
1418    table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
1419  File "pyarrow/table.pxi", line 3700, in pyarrow.lib.Table.from_pylist
1420  File "pyarrow/table.pxi", line 5221, in pyarrow.lib._from_pylist
1421  File "pyarrow/table.pxi", line 3575, in pyarrow.lib.Table.from_arrays
1422  File "pyarrow/table.pxi", line 1383, in pyarrow.lib._sanitize_arrays
1423  File "pyarrow/table.pxi", line 1364, in pyarrow.lib._schema_from_arrays
1424  File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
1425  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
1426  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
1427  File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status
1428pyarrow.lib.ArrowTypeError: Expected bytes, got a 'int' object{code}
 
{code:java}
import numpy as np

pdf = self._to_pandas()
types = pdf.dtypes
self.assertEqual(types[0], np.int32)
self.assertEqual(types[1], np.object)
self.assertEqual(types[2], np.bool)
self.assertEqual(types[3], np.float32)
self.assertEqual(types[4], np.object)  # datetime.date
self.assertEqual(types[5], "datetime64[ns]")
self.assertEqual(types[6], "datetime64[ns]")
self.assertEqual(types[7], "timedelta64[ns]") {code}
{code:java}
Traceback (most recent call last): 1434 File "/__w/spark/spark/python/pyspark/sql/tests/test_dataframe.py", line 1039, in test_to_pandas 1435 self.assertEqual(types[5], "datetime64[ns]") 1436AssertionError: datetime64[ns, Etc/UTC] != 'datetime64[ns]' 1437
{code}
Bug: Protect null `SQLExecutionUIData#description` in `SQLExecutionUIDataSerializer`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/23 07:10
Affects Version/s: 3.4.0

export LIVE_UI_LOCAL_STORE_DIR = /tmp/spark-ui

mvn clean install -pl sql/core -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest -Dtest=none -DwildcardSuites=org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOff -am 

 

no test failed, but some error message:

 
{code:java}
14:46:44.514 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener SQLAppStatusListener threw an exception
java.lang.NullPointerException
    at org.apache.spark.status.protobuf.StoreTypes$SQLExecutionUIData$Builder.setDescription(StoreTypes.java:46500)
    at org.apache.spark.status.protobuf.sql.SQLExecutionUIDataSerializer.serialize(SQLExecutionUIDataSerializer.scala:34)
    at org.apache.spark.status.protobuf.sql.SQLExecutionUIDataSerializer.serialize(SQLExecutionUIDataSerializer.scala:28)
    at org.apache.spark.status.protobuf.KVStoreProtobufSerializer.serialize(KVStoreProtobufSerializer.scala:30)
    at org.apache.spark.util.kvstore.RocksDB.write(RocksDB.java:188)
    at org.apache.spark.status.ElementTrackingStore.write(ElementTrackingStore.scala:123)
    at org.apache.spark.status.ElementTrackingStore.write(ElementTrackingStore.scala:127)
    at org.apache.spark.status.LiveEntity.write(LiveEntity.scala:50)
    at org.apache.spark.sql.execution.ui.SQLAppStatusListener.update(SQLAppStatusListener.scala:456)
    at org.apache.spark.sql.execution.ui.SQLAppStatusListener.onJobStart(SQLAppStatusListener.scala:124)
    at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37)
    at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
    at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
    at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
    at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
    at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
    at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
    at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
    at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
    at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
    at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
    at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1444)
    at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
14:46:44.936 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener SQLAppStatusListener threw an exception {code}
Bug: Using checkpoints in Spark Structured Streaming with the foreachBatch sink
Status: Open | Priority: Major | Resolution: 
Created: 17/Jan/23 14:48
Affects Version/s: 3.3.1

I want to build a fault-tolerant, recoverable Spark job (using Structured Streaming in PySpark) that reads a data stream from Kafka and uses the [{{foreachBatch}}|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#foreachbatch] sink to implement a stateful transformation before writing the resulting data to the actual sink.

The basic structure of my Spark job is like this:
{code}
counter = 0

def batch_handler(df, batch_id):
  global counter
  counter += 1
  df.withColumn('counter', lit(counter)).show(truncate=30)

spark = (SparkSession.builder
  .appName('test.stateful.checkpoint')
  .config('spark.jars.packages', f'{KAFKA_SQL},{KAFKA_CLNT}')
  .getOrCreate())

source = (spark.readStream
  .format('kafka')
  .options(**KAFKA_OPTIONS)
  .option('subscribe', 'topic-spark-stateful')
  .option('startingOffsets', 'earliest')
  .option('includeHeaders', 'true')
  .load())

(source
  .selectExpr('CAST(value AS STRING) AS data', 'CAST(timestamp AS STRING) AS time')
  .writeStream
  .option('checkpointLocation', './checkpoints/stateful')
  .foreachBatch(batch_handler)
  .start()
  .awaitTermination())
{code}

where the simplified {{batch_handler}} function is a stand-in for the stateful transformation + writer to the actual data sink. Also for simplicity I am using a local folder as checkpoint location. 

This works fine as far as checkpointing of Kafka offsets is concerned. But how can I include the state of my custom batch handler ({{counter}} in my simplified example) in the checkpoints such that the job can pick up where it left after a crash?

The [Spark Structured Streaming Guide|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing] doesn't say anything on the topic. With the [{{foreach}}|(https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#foreach] sink I can pass a custom row handler object but this seems to support only {{open}}, {{process}}, and {{close}} methods.

Would it make sense to create a "Request" or even "Feature" ticket to enhance this with methods for restoring state from a checkpoint and exporting state to support checkpointing?

PS: I have posted this on [SOF|https://stackoverflow.com/questions/74864425], too. If anyone cares to answer or comment I'd be happy to upvote their post.
Bug: Different result in nested lambda function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/23 11:32
Affects Version/s: 3.4.0

test_nested_higher_order_function


{code:java}
Traceback (most recent call last):
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/tests/test_functions.py", line 814, in test_nested_higher_order_function
    self.assertEquals(actual, expected)
AssertionError: Lists differ: [Row(n='a', l='a'), Row(n='b', l='b'), Row[124 chars]'c')] != [(1, 'a'), (1, 'b'), (1, 'c'), (2, 'a'), ([43 chars]'c')]

First differing element 0:
Row(n='a', l='a')
(1, 'a')

- [Row(n='a', l='a'),
-  Row(n='b', l='b'),
-  Row(n='c', l='c'),
-  Row(n='a', l='a'),
-  Row(n='b', l='b'),
-  Row(n='c', l='c'),
-  Row(n='a', l='a'),
-  Row(n='b', l='b'),
-  Row(n='c', l='c')]
+ [(1, 'a'),
+  (1, 'b'),
+  (1, 'c'),
+  (2, 'a'),
+  (2, 'b'),
+  (2, 'c'),
+  (3, 'a'),
+  (3, 'b'),
+  (3, 'c')]


{code}
Bug: Make `createDataFrame` support array
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 02:34
Affects Version/s: 3.4.0

{code:java}
import array

data = [Row(longarray=array.array("l", [-9223372036854775808, 0, 9223372036854775807]))]
df = self.spark.createDataFrame(data) {code}
Error:
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 1220, in test_create_dataframe_from_array_of_long
    df = self.spark.createDataFrame(data)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", line 260, in createDataFrame
    table = pa.Table.from_pylist([row.asDict(recursive=True) for row in _data])
  File "pyarrow/table.pxi", line 3700, in pyarrow.lib.Table.from_pylist
  File "pyarrow/table.pxi", line 5221, in pyarrow.lib._from_pylist
  File "pyarrow/table.pxi", line 3575, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 1383, in pyarrow.lib._sanitize_arrays
  File "pyarrow/table.pxi", line 1364, in pyarrow.lib._schema_from_arrays
  File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert array('l', [-9223372036854775808, 0, 9223372036854775807]) with type array.array: did not recognize Python value type when inferring an Arrow data type{code}
Bug: Harden SQLSTATE usage for error classes (2)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/23 21:16
Affects Version/s: 3.4.0

Error classes are great, but for JDBC, ODBC etc the SQLSTATEs of the standard reign.
We have started adding SQLSTATEs but have not really paid attention to their correctness.
Follow up to: https://issues.apache.org/jira/browse/SPARK-41994
Bug: Parity in String representation of higher_order_function's output
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:46
Affects Version/s: 3.4.0

{code:java}
from pyspark.sql.functions import flatten, struct, transform

df = self.spark.sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")

actual = df.select(
    flatten(
        transform(
            "numbers",
            lambda number: transform(
                "letters", lambda letter: struct(number.alias("n"), letter.alias("l"))
            ),
        )
    )
).first()[0]

expected = [
    (1, "a"),
    (1, "b"),
    (1, "c"),
    (2, "a"),
    (2, "b"),
    (2, "c"),
    (3, "a"),
    (3, "b"),
    (3, "c"),
]

self.assertEquals(actual, expected){code}
{code:java}
    Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 809, in test_nested_higher_order_function
    self.assertEquals(actual, expected)
AssertionError: Lists differ: [{'n': 'a', 'l': 'a'}, {'n': 'b', 'l': 'b'[151 chars]'c'}] != [(1, 'a'), (1, 'b'), (1, 'c'), (2, 'a'), ([43 chars]'c')]

First differing element 0:
{'n': 'a', 'l': 'a'}
(1, 'a')

- [{'l': 'a', 'n': 'a'},
-  {'l': 'b', 'n': 'b'},
-  {'l': 'c', 'n': 'c'},
-  {'l': 'a', 'n': 'a'},
-  {'l': 'b', 'n': 'b'},
-  {'l': 'c', 'n': 'c'},
-  {'l': 'a', 'n': 'a'},
-  {'l': 'b', 'n': 'b'},
-  {'l': 'c', 'n': 'c'}]
+ [(1, 'a'),
+  (1, 'b'),
+  (1, 'c'),
+  (2, 'a'),
+  (2, 'b'),
+  (2, 'c'),
+  (3, 'a'),
+  (3, 'b'),
+  (3, 'c')]
{code}
Bug: createDataFrame with array.array
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 05:53
Affects Version/s: 3.4.0

{code}
pyspark/sql/tests/test_types.py:964 (TypesParityTests.test_array_types)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_array_types>

    def test_array_types(self):
        # This test need to make sure that the Scala type selected is at least
        # as large as the python's types. This is necessary because python's
        # array types depend on C implementation on the machine. Therefore there
        # is no machine independent correspondence between python's array types
        # and Scala types.
        # See: https://docs.python.org/2/library/array.html
    
        def assertCollectSuccess(typecode, value):
            row = Row(myarray=array.array(typecode, [value]))
            df = self.spark.createDataFrame([row])
            self.assertEqual(df.first()["myarray"][0], value)
    
        # supported string types
        #
        # String types in python's array are "u" for Py_UNICODE and "c" for char.
        # "u" will be removed in python 4, and "c" is not supported in python 3.
        supported_string_types = []
        if sys.version_info[0] < 4:
            supported_string_types += ["u"]
            # test unicode
>           assertCollectSuccess("u", "a")

../test_types.py:986: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../test_types.py:975: in assertCollectSuccess
    df = self.spark.createDataFrame([row])
../../connect/session.py:278: in createDataFrame
    _table = pa.Table.from_pylist([row.asDict(recursive=True) for row in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert array('u', 'a') with type array.array: did not recognize Python value type when inferring an Arrow data type

pyarrow/error.pxi:100: ArrowInvalid

{code}
Bug: Harden SQLSTATE usage for error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/23 21:39
Affects Version/s: 3.4.0

Error classes are great, but for JDBC, ODBC etc the SQLSTATEs of the standard reign.
We have started adding SQLSTATEs but have not really paid attention to their correctness.
Here is a unified view of SQLSTATE's used in the  [Industry.|https://docs.google.com/spreadsheets/d/1hrQBSuHooiozUNAQTHiYq3WidS1uliHpl9cYfWpig1c/edit?usp=sharing]
Bug: Fix map_filter and map_zip_with output order
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/23 15:19
Affects Version/s: 3.4.0


{code:java}
File "/Users/jiaan.geng/git-local/github-forks/spark/python/pyspark/sql/connect/functions.py", line 1423, in pyspark.sql.connect.functions.map_filter
Failed example:
    df.select(map_filter(
        "data", lambda _, v: v > 30.0).alias("data_filtered")
    ).show(truncate=False)
Expected:
    +--------------------------+
    |data_filtered             |
    +--------------------------+
    |{baz -> 32.0, foo -> 42.0}|
    +--------------------------+
Got:
    +--------------------------+
    |data_filtered             |
    +--------------------------+
    |{foo -> 42.0, baz -> 32.0}|
    +--------------------------+
    <BLANKLINE>
**********************************************************************
File "/Users/jiaan.geng/git-local/github-forks/spark/python/pyspark/sql/connect/functions.py", line 1465, in pyspark.sql.connect.functions.map_zip_with
Failed example:
    df.select(map_zip_with(
        "base", "ratio", lambda k, v1, v2: round(v1 * v2, 2)).alias("updated_data")
    ).show(truncate=False)
Expected:
    +---------------------------+
    |updated_data               |
    +---------------------------+
    |{SALES -> 16.8, IT -> 48.0}|
    +---------------------------+
Got:
    +---------------------------+
    |updated_data               |
    +---------------------------+
    |{IT -> 48.0, SALES -> 16.8}|
    +---------------------------+
    <BLANKLINE>
{code}
Bug: Map data show in different order
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 14:57
Affects Version/s: 3.4.0

not sure whether this needs to be fixed:


{code:java}
**********************************************************************
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1623, in pyspark.sql.connect.functions.transform_keys
Failed example:
    df.select(transform_keys(
        "data", lambda k, _: upper(k)).alias("data_upper")
    ).show(truncate=False)
Expected:
    +-------------------------+
    |data_upper               |
    +-------------------------+
    |{BAR -> 2.0, FOO -> -2.0}|
    +-------------------------+
Got:
    +-------------------------+
    |data_upper               |
    +-------------------------+
    |{FOO -> -2.0, BAR -> 2.0}|
    +-------------------------+
    <BLANKLINE>
**********************************************************************
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1630, in pyspark.sql.connect.functions.transform_values
Failed example:
    df.select(transform_values(
        "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v)
    ).alias("new_data")).show(truncate=False)
Expected:
    +---------------------------------------+
    |new_data                               |
    +---------------------------------------+
    |{OPS -> 34.0, IT -> 20.0, SALES -> 2.0}|
    +---------------------------------------+
Got:
    +---------------------------------------+
    |new_data                               |
    +---------------------------------------+
    |{IT -> 20.0, SALES -> 2.0, OPS -> 34.0}|
    +---------------------------------------+
    <BLANKLINE>
**********************************************************************
   1 of   2 in pyspark.sql.connect.functions.transform_keys
   1 of   2 in pyspark.sql.connect.functions.transform_values

{code}
Bug: Support data type ndarray
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:47
Affects Version/s: 3.4.0

{code:java}
import numpy as np

arr_dtype_to_spark_dtypes = [
    ("int8", [("b", "array<smallint>")]),
    ("int16", [("b", "array<smallint>")]),
    ("int32", [("b", "array<int>")]),
    ("int64", [("b", "array<bigint>")]),
    ("float32", [("b", "array<float>")]),
    ("float64", [("b", "array<double>")]),
]
for t, expected_spark_dtypes in arr_dtype_to_spark_dtypes:
    arr = np.array([1, 2]).astype(t)
    self.assertEqual(
        expected_spark_dtypes, self.spark.range(1).select(lit(arr).alias("b")).dtypes
    )
arr = np.array([1, 2]).astype(np.uint)
with self.assertRaisesRegex(
    TypeError, "The type of array scalar '%s' is not supported" % arr.dtype
):
    self.spark.range(1).select(lit(arr).alias("b")){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 1100, in test_ndarray_input
    expected_spark_dtypes, self.spark.range(1).select(lit(arr).alias("b")).dtypes
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/utils.py", line 332, in wrapped
    return getattr(functions, f.__name__)(*args, **kwargs)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 198, in lit
    return Column(LiteralExpression._from_value(col))
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/expressions.py", line 266, in _from_value
    return LiteralExpression(value=value, dataType=LiteralExpression._infer_type(value))
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/expressions.py", line 262, in _infer_type
    raise ValueError(f"Unsupported Data Type {type(value).__name__}")
ValueError: Unsupported Data Type ndarray {code}
Bug: Implement DataFrameReader.orc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 05:09
Affects Version/s: 3.4.0

{code}

pyspark/sql/tests/test_datasources.py:114 (DataSourcesParityTests.test_read_multiple_orc_file)
self = <pyspark.sql.tests.connect.test_parity_datasources.DataSourcesParityTests testMethod=test_read_multiple_orc_file>

    def test_read_multiple_orc_file(self):
>       df = self.spark.read.orc(
            [
                "python/test_support/sql/orc_partitioned/b=0/c=0",
                "python/test_support/sql/orc_partitioned/b=1/c=1",
            ]
        )

../test_datasources.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyspark.sql.connect.readwriter.DataFrameReader object at 0x7fb170946b50>
args = (['python/test_support/sql/orc_partitioned/b=0/c=0', 'python/test_support/sql/orc_partitioned/b=1/c=1'],)
kwargs = {}

    def orc(self, *args: Any, **kwargs: Any) -> None:
>       raise NotImplementedError("orc() is not implemented.")
E       NotImplementedError: orc() is not implemented.

../../connect/readwriter.py:228: NotImplementedError

{code}
Bug: Implement DataFrameReader.csv
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 05:05
Affects Version/s: 3.4.0

{code}
                                                                          
pyspark/sql/tests/test_datasources.py:147 (DataSourcesParityTests.test_checking_csv_header)
self = <pyspark.sql.tests.connect.test_parity_datasources.DataSourcesParityTests testMethod=test_checking_csv_header>

    def test_checking_csv_header(self):
        path = tempfile.mkdtemp()
        shutil.rmtree(path)
        try:
            self.spark.createDataFrame([[1, 1000], [2000, 2]]).toDF("f1", "f2").write.option(
                "header", "true"
            ).csv(path)
            schema = StructType(
                [
                    StructField("f2", IntegerType(), nullable=True),
                    StructField("f1", IntegerType(), nullable=True),
                ]
            )
            df = (
>               self.spark.read.option("header", "true")
                .schema(schema)
                .csv(path, enforceSchema=False)
            )

../test_datasources.py:162: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyspark.sql.connect.readwriter.DataFrameReader object at 0x7fb118289520>
args = ('/var/folders/0c/q8y15ybd3tn7sr2_jmbmftr80000gp/T/tmp4kdxohcw',)
kwargs = {'enforceSchema': False}

    def csv(self, *args: Any, **kwargs: Any) -> None:
>       raise NotImplementedError("csv() is not implemented.")
E       NotImplementedError: csv() is not implemented.

../../connect/readwriter.py:225: NotImplementedError

{code}