av9ash/GitBugs huggingface dataset:
Bug: Interpreted mode subexpression elimination can throw exception during insert
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/23 19:16
Affects Version/s: 3.3.1

Example:
{noformat}
drop table if exists tbl1;
create table tbl1 (a int, b int) using parquet;

set spark.sql.codegen.wholeStage=false;
set spark.sql.codegen.factoryMode=NO_CODEGEN;

insert into tbl1
select id as a, id as b
from range(1, 5);
{noformat}
This results in the following exception:
{noformat}
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.ExpressionProxy cannot be cast to org.apache.spark.sql.catalyst.expressions.Cast
	at org.apache.spark.sql.catalyst.expressions.CheckOverflowInTableInsert.withNewChildInternal(Cast.scala:2514)
	at org.apache.spark.sql.catalyst.expressions.CheckOverflowInTableInsert.withNewChildInternal(Cast.scala:2512)
{noformat}
The query produces 2 bigint values, but the table's schema expects 2 int values, so Spark wraps each output field with a {{Cast}}.

Later, in {{InterpretedUnsafeProjection}}, {{prepareExpressions}} tries to wrap the two {{Cast}} expressions with an {{ExpressionProxy}}. However, the parent expression of each {{Cast}} is a {{CheckOverflowInTableInsert}} expression, which does not accept {{ExpressionProxy}} as a child.
Bug: Implement DataFrameWriterV2 (ReadwriterV2Tests)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 02:49
Affects Version/s: 3.4.0

{code}
pyspark/sql/tests/test_readwriter.py:182 (ReadwriterV2ParityTests.test_api)
self = <pyspark.sql.tests.connect.test_parity_readwriter.ReadwriterV2ParityTests testMethod=test_api>

    def test_api(self):
        df = self.df
>       writer = df.writeTo("testcat.t")

../test_readwriter.py:185: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = DataFrame[key: bigint, value: string], args = ('testcat.t',), kwargs = {}

    def writeTo(self, *args: Any, **kwargs: Any) -> None:
>       raise NotImplementedError("writeTo() is not implemented.")
E       NotImplementedError: writeTo() is not implemented.

../../connect/dataframe.py:1529: NotImplementedError
{code}
Bug: DataFrame.join: ambiguous column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 08:19
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/column.py", line 106, in pyspark.sql.connect.column.Column.eqNullSafe
Failed example:
    df1.join(df2, df1["value"] == df2["value"]).count()
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.column.Column.eqNullSafe[4]>", line 1, in <module>
        df1.join(df2, df1["value"] == df2["value"]).count()
      File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 151, in count
        pdd = self.agg(_invoke_function("count", lit(1))).toPandas()
      File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 1031, in toPandas
        return self._session.client.to_pandas(query)
      File "/.../spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/.../spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/.../spark/python/pyspark/sql/connect/client.py", line 619, in _handle_error
        raise SparkConnectAnalysisException(
    pyspark.sql.connect.client.SparkConnectAnalysisException: [AMBIGUOUS_REFERENCE] Reference `value` is ambiguous, could be: [`value`, `value`].
{code}
Bug: Parity in String representation of Column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:40
Affects Version/s: 3.4.0

{code:java}
from pyspark.sql import functions

funs = [
    (functions.acosh, "ACOSH"),
    (functions.asinh, "ASINH"),
    (functions.atanh, "ATANH"),
]

cols = ["a", functions.col("a")]

for f, alias in funs:
    for c in cols:
        self.assertIn(f"{alias}(a)", repr(f(c))){code}
{code:java}
 Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 271, in test_inverse_trig_functions
    self.assertIn(f"{alias}(a)", repr(f(c)))
AssertionError: 'ACOSH(a)' not found in "Column<'acosh(ColumnReference(a))'>"{code}
 

 
{code:java}
from pyspark.sql.functions import col, lit, overlay
from itertools import chain
import re

actual = list(
    chain.from_iterable(
        [
            re.findall("(overlay\\(.*\\))", str(x))
            for x in [
                overlay(col("foo"), col("bar"), 1),
                overlay("x", "y", 3),
                overlay(col("x"), col("y"), 1, 3),
                overlay("x", "y", 2, 5),
                overlay("x", "y", lit(11)),
                overlay("x", "y", lit(2), lit(5)),
            ]
        ]
    )
)

expected = [
    "overlay(foo, bar, 1, -1)",
    "overlay(x, y, 3, -1)",
    "overlay(x, y, 1, 3)",
    "overlay(x, y, 2, 5)",
    "overlay(x, y, 11, -1)",
    "overlay(x, y, 2, 5)",
]

self.assertListEqual(actual, expected)

df = self.spark.createDataFrame([("SPARK_SQL", "CORE", 7, 0)], ("x", "y", "pos", "len"))

exp = [Row(ol="SPARK_CORESQL")]
self.assertTrue(
    all(
        [
            df.select(overlay(df.x, df.y, 7, 0).alias("ol")).collect() == exp,
            df.select(overlay(df.x, df.y, lit(7), lit(0)).alias("ol")).collect() == exp,
            df.select(overlay("x", "y", "pos", "len").alias("ol")).collect() == exp,
        ]
    )
) {code}
{code:java}
    Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 675, in test_overlay
    self.assertListEqual(actual, expected)
AssertionError: Lists differ: ['overlay(ColumnReference(foo), ColumnReference(bar[402 chars]5))'] != ['overlay(foo, bar, 1, -1)', 'overlay(x, y, 3, -1)'[90 chars] 5)']

First differing element 0:
'overlay(ColumnReference(foo), ColumnReference(bar), Literal(1), Literal(-1))'
'overlay(foo, bar, 1, -1)'

- ['overlay(ColumnReference(foo), ColumnReference(bar), Literal(1), Literal(-1))',
-  'overlay(ColumnReference(x), ColumnReference(y), Literal(3), Literal(-1))',
-  'overlay(ColumnReference(x), ColumnReference(y), Literal(1), Literal(3))',
-  'overlay(ColumnReference(x), ColumnReference(y), Literal(2), Literal(5))',
-  'overlay(ColumnReference(x), ColumnReference(y), Literal(11), Literal(-1))',
-  'overlay(ColumnReference(x), ColumnReference(y), Literal(2), Literal(5))']
+ ['overlay(foo, bar, 1, -1)',
+  'overlay(x, y, 3, -1)',
+  'overlay(x, y, 1, 3)',
+  'overlay(x, y, 2, 5)',
+  'overlay(x, y, 11, -1)',
+  'overlay(x, y, 2, 5)']
 {code}
Bug: SparkSession.read support reading with schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 15:14
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/readwriter.py", line 122, in pyspark.sql.connect.readwriter.DataFrameReader.load
Failed example:
    with tempfile.TemporaryDirectory() as d:
        # Write a DataFrame into a CSV file with a header
        df = spark.createDataFrame([{"age": 100, "name": "Hyukjin Kwon"}])
        df.write.option("header", True).mode("overwrite").format("csv").save(d)

        # Read the CSV file as a DataFrame with 'nullValue' option set to 'Hyukjin Kwon',
        # and 'header' option set to `True`.
        df = spark.read.load(
            d, schema=df.schema, format="csv", nullValue="Hyukjin Kwon", header=True)
        df.printSchema()
        df.show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.readwriter.DataFrameReader.load[1]>", line 10, in <module>
        df.printSchema()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1039, in printSchema
        print(self._tree_string())
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1035, in _tree_string
        query = self._plan.to_proto(self._session.client)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/plan.py", line 92, in to_proto
        plan.root.CopyFrom(self.plan(session))
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/plan.py", line 245, in plan
        plan.read.data_source.schema = self.schema
    TypeError: bad argument type for built-in operation {code}
Bug: DataFrame.createOrReplaceGlobalTempView - SparkConnectException: requirement failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 16:46
Affects Version/s: 3.4.0

{code:java}
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])
>>> df2 = df.filter(df.age > 3)
>>> df2.createOrReplaceGlobalTempView("people") {code}
{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1292, in pyspark.sql.connect.dataframe.DataFrame.createOrReplaceGlobalTempView
Failed example:
    df2.createOrReplaceGlobalTempView("people")
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.createOrReplaceGlobalTempView[3]>", line 1, in <module>
        df2.createOrReplaceGlobalTempView("people")
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1192, in createOrReplaceGlobalTempView
        self._session.client.execute_command(command)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 459, in execute_command
        self._execute(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 547, in _execute
        self._handle_error(rpc_error)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 625, in _handle_error
        raise SparkConnectException(status.message) from None
    pyspark.sql.connect.client.SparkConnectException: requirement failed 

{code}
Bug: Filtering by composite field name like `field name` doesn't work with pushDownPredicate = true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/23 19:08
Affects Version/s: 3.3.0

Suppose we have some table in postgresql with field `Last Name` The following code results in error

Dataset<Row> dataset = sparkSession.read()
.format("jdbc")
.option("url", myUrl)
.option("dbtable", "myTable")
.option("user", "myUser")
.option("password", "muPassword")
.load();

dataset.where("`Last Name`='Tessel'").show();    //error

 

 

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
Syntax error at or near 'Name': extra input 'Name'(line 1, pos 5)

== SQL ==
Last Name
-----^^^

    at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseMultipartIdentifier(ParseDriver.scala:67)
    at org.apache.spark.sql.connector.expressions.LogicalExpressions$.parseReference(expressions.scala:40)
    at org.apache.spark.sql.connector.expressions.FieldReference$.apply(expressions.scala:368)
    at org.apache.spark.sql.sources.IsNotNull.toV2(filters.scala:262)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.$anonfun$unhandledFilters$1(JDBCRelation.scala:278)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.$anonfun$unhandledFilters$1$adapted(JDBCRelation.scala:278)

 

But if we set pushDownPredicate to false everything works fine.
Bug: Different exception message in DataFrame.unpivot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/23 09:25
Affects Version/s: 3.4.0

Running {{test_parity_dataframe DataFrameParityTests.test_unpivot_negative}} fails as below:

{code}
        with self.subTest(desc="with no value columns"):
            for values in [[], ()]:
                with self.subTest(values=values):
                    with self.assertRaisesRegex(
                        Exception,  # (AnalysisException, SparkConnectException)
                        r".*\[UNPIVOT_REQUIRES_VALUE_COLUMNS] At least one value column "
                        r"needs to be specified for UNPIVOT, all columns specified as ids.*",
                    ):
>                       df.unpivot("id", values, "var", "val").collect()
E                       AssertionError: ".*\[UNPIVOT_REQUIRES_VALUE_COLUMNS] At least one value column needs to be specified for UNPIVOT, all columns specified as ids.*" does not match "[UNPIVOT_VALUE_DATA_TYPE_MISMATCH] Unpivot value columns must share a least common type, some types do not: ["BIGINT" (`int`), "DOUBLE" (`double`), "STRING" (`str`)]
E                       Plan: 'Unpivot ArraySeq(id#2947L), List(List(int#2948L), List(double#2949), List(str#2950)), var, [val]
E                       +- Project [id#2939L AS id#2947L, int#2940L AS int#2948L, double#2941 AS double#2949, str#2942 AS str#2950]
E                          +- LocalRelation [id#2939L, int#2940L, double#2941, str#2942]
E                       "
{code}
Bug: Spark 3.3 Backport: SPARK-41344 Reading V2 datasource masks underlying error
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Jan/23 07:12
Affects Version/s: 3.3.0

The underlying error message is thrown away, leading to a misleading/non-user friendly error message.

Full description in https://issues.apache.org/jira/browse/SPARK-41344. I will backport this to Spark 3.3
Bug: Memory usage on executors increased drastically for a complex query with large number of addition operations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/23 20:43
Affects Version/s: 3.3.0

With the [recent changes|https://github.com/apache/spark/pull/37851]  in the expression canonicalization, a complex query with a large number of Add operations ends up consuming 10x more memory on the executors.

The reason for this issue is that with the new changes the canonicalization process ends up generating lot of intermediate objects, especially for complex queries with a large number of commutative operators. In this specific case, a heap histogram analysis shows that a large number of Add objects use the extra memory.
This issue does not happen before PR [#37851.|https://github.com/apache/spark/pull/37851]

The high memory usage causes the executors to lose heartbeat signals and results in task failures.
Bug: Extract the function that construct the select statement for JDBC dialect.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/23 09:03
Affects Version/s: 3.4.0

Currently, JDBCRDD uses fixed format for SELECT statement.


{code:java}
val sqlText = options.prepareQuery +
      s"SELECT $columnList FROM ${options.tableOrQuery} $myTableSampleClause" +
      s" $myWhereClause $getGroupByClause $getOrderByClause $myLimitClause $myOffsetClause"
{code}


But some databases have different syntax that uses different keyword or sort. For example, MS SQL Server uses keyword TOP to describe LIMIT clause or Top N.
The LIMIT clause of MS SQL Server.


{code:java}
SELECT TOP(1) Model, Color, Price  
      FROM dbo.Cars  
      WHERE Color = 'blue'
{code}


The Top N of MS SQL Server.


{code:java}
SELECT TOP(1) Model, Color, Price  
FROM dbo.Cars  
WHERE Color = 'blue'  
ORDER BY Price ASC
{code}
Bug: createDataFrame should corse types of string float to float
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 12/Jan/23 06:05
Affects Version/s: 3.4.0

{code}

pyspark/sql/tests/test_types.py:245 (TypesParityTests.test_infer_schema_upcast_float_to_string)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_upcast_float_to_string>

    def test_infer_schema_upcast_float_to_string(self):
>       df = self.spark.createDataFrame([[1.33, 1], ["2.1", 1]], schema=["a", "b"])

../test_types.py:247: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:282: in createDataFrame
    _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert '2.1' with type str: tried to convert to double

pyarrow/error.pxi:100: ArrowInvalid
{code}
Bug: SPJ: Support partially clustered distribution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/23 00:20
Affects Version/s: 3.3.1

Currently the storage-partitioned join requires both sides to be fully clustered on the partition values, that is, all input partitions reported by a V2 data source shall be grouped by partition values before the join happens. This could lead to data skew issues if a particular partition value is associated with a large amount of rows.

 

To combat this, we can introduce the idea of partially clustered distribution, which means that only one side of the join is required to be fully clustered, while the other side is not. This allows Spark to increase the parallelism of the join and avoid the data skewness.
Bug: Push down limit through Python UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/23 11:52
Affects Version/s: 3.4.0

{code}
from pyspark.sql.functions import udf

spark.range(10).write.mode("overwrite").parquet("/tmp/abc")

@udf(returnType='string')
def my_udf(arg):
    return arg


df = spark.read.parquet("/tmp/abc")
df.limit(10).withColumn("prediction", my_udf(df["id"])).explain()
{code}

As an example. since Python UDFs are executed asynchronously, so pushing limits benefit the performance.

{code}
== Physical Plan ==
CollectLimit 10
+- *(2) Project [id#3L, pythonUDF0#10 AS prediction#6]
   +- BatchEvalPython [my_udf(id#3L)#5], [pythonUDF0#10]
      +- *(1) ColumnarToRow
         +- FileScan parquet [id#3L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/abc], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint>
{code}

This is a regression from Spark 3.3.1:

{code}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [id#3L, pythonUDF0#10 AS prediction#6]
   +- BatchEvalPython [my_udf(id#3L)#5], [pythonUDF0#10]
      +- GlobalLimit 10
         +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=30]
            +- LocalLimit 10
               +- FileScan parquet [id#3L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/abc], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint>
{code}
Bug: Improve UNSUPPORTED_DATA_TYPE message for complex types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/23 03:43
Affects Version/s: 3.4.0

spark-sql> SELECT CAST(array(1, 2, 3) AS ARRAY);

[UNSUPPORTED_DATATYPE] Unsupported data type "ARRAY"(line 1, pos 30)

== SQL ==

SELECT CAST(array(1, 2, 3) AS ARRAY)

------------------------------^^^

This error message is confusing. We support ARRAY. We just require it to be typed.
We should have an error like:
[INCOMPLETE_TYPE_DEFINITION.ARRAY] The definition of type `ARRAY` is incomplete. You must provide an element type. For example: `ARRAY\<INT\>`.
Similarly for STRUCT and MAP.
Bug: Table schema changes via V2SessionCatalog with HiveExternalCatalog
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Jan/23 19:50
Affects Version/s: 3.5.0

When configuring HiveExternalCatalog there are certain schema change operations in V2SessionCatalog that are not performed due to the use of the alterTable method instead of alterTableDataSchema.
Bug: SPIP: Lazy Materialization for Parquet Read Performance Improvement
Status: Open | Priority: Major | Resolution: 
Created: 31/Jan/23 09:41
Affects Version/s: 3.5.0

    Spark-SQL filter operation is a common workload in order to select specific rows from persisted data. The current implementation of Spark requires the read values to materialize (i.e. de-compress, de-code, etc...) onto memory first before applying the filters. This approach means that the filters may eventually throw away many values, resulting in wasted computations. Alternatively, evaluating the filters first and lazily materializing only the used values can save waste and improve the read performance. Lazy materialization has been employed by other distributed SQL engines such as Velox and Presto/Trino, but this approach has not yet been extended to Spark with Parquet.

SPIP: https://docs.google.com/document/d/1Kr3y2fVZUbQXGH0y8AvdCAeWC49QJjpczapiaDvFzME
Bug: Forbid deicmal type if precision less than 1
Status: In Progress | Priority: Major | Resolution: 
Created: 31/Jan/23 04:24
Affects Version/s: 3.5.0

Spark does not actually support decimal type with 0 precision. e.g.

 
{code:java}
– work with in-memory catalog
create table t (c decimal(0, 0)) using parquet;

-- fail with parquet
-- java.lang.IllegalArgumentException: Invalid DECIMAL precision: 0
-- at org.apache.parquet.Preconditions.checkArgument(Preconditions.java:57)
insert into table t values(0);

-- fail with hive catalog
-- Caused by: java.lang.IllegalArgumentException: Decimal precision out of allowed range [1,38]
-- at org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils.validateParameter(HiveDecimalUtils.java:44)
create table t (c decimal(0, 0)) using parquet;{code}
Bug: createDataFrame should corse types of string false to bool false
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/23 06:04
Affects Version/s: 3.4.0

{code}

pyspark/sql/tests/test_types.py:249 (TypesParityTests.test_infer_schema_upcast_boolean_to_string)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_upcast_boolean_to_string>

    def test_infer_schema_upcast_boolean_to_string(self):
>       df = self.spark.createDataFrame([[True, 1], ["false", 1]], schema=["a", "b"])

../test_types.py:251: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:282: in createDataFrame
    _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert 'false' with type str: tried to convert to boolean

pyarrow/error.pxi:100: ArrowInvalid

{code}