av9ash/GitBugs huggingface dataset:
Bug: Adding State Schema ID to State Rows to schema evolution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/24 18:05
Affects Version/s: 4.0.0

In order to support schema evolution without a full rewrite, we are prepending each state row with a state schema ID, which we will use in subsequent changes to look up the schema that a row was written with. This change adds the encoding and decoding of this prefix through the StateRowPrefixEncoder.
Bug: Allow long values for timestamp add expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Dec/24 14:13
Affects Version/s: 4.0.0

Currently executing the following query fails because of cast overflow:


{code:sql}
select
  timestampadd(
    millisecond,
    1733011227403,
    timestamp '1970-01-01T00:00:00.000Z'
  )
{code}

Signature of this function should be changed to allow long values.
Bug: DELETE fails with INTERNAL_ERROR on tables with CHAR(N) columns 
Status: Open | Priority: Major | Resolution: 
Created: 26/Dec/24 23:52
Affects Version/s: 3.5.4

Repro (SQL):
{code:java}
create table test (a char(5)) using parquet;
delete from test; {code}
Error:
{noformat}
org.apache.spark.SparkException: [INTERNAL_ERROR] Unexpected table relation: Project [staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, a#0, 5, true, false, true) AS a#1]                    +- HiveTableRelation [`spark_catalog`.`default`.`test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [a#0], Partition Cols: []]
        at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
        at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
        at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Strategy.apply(DataSourceV2Strategy.scala:326)
...{noformat}
Single-line Docker command to repro the issue:
{code:java}
docker run -it --rm apache/spark /bin/bash -c '/opt/spark/bin/spark-sql -f <(echo "create table test (a char(5)); delete from test;")' {code}
Bug: Support ampersand in spark-submit for windows
Status: Open | Priority: Major | Resolution: 
Created: 24/Dec/24 13:48
Affects Version/s: 3.5.4

The spark-submit.cmd does not support passing `-conf` flags that contain ampersand (&) this issue can be easly manifested when configuring urls (for example, defining the `
spark.mongodb.read.connection.uri` for the mongo connector)
Bug: Move spark-sql related classes to sql/hive
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 20/Dec/24 10:54
Affects Version/s: 3.5.3

Sometimes we only use spark-sql, but not spark-thriftserver. So I think we should move the spark-sql related classes to sql/hive. For such users, we don't need to specify -Phive-thriftserver for compilation.
Bug: BlockMissingException should be thrown even if ignoreCorruptFiles is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/24 00:31
Affects Version/s: 3.5.3


{noformat}
24/11/29 01:56:00 WARN FileScanRDD: Skipped the rest of the content in the corrupted file: path: viewfs://hadoop-cluster/path/to/data/part-00320-7915e327-3214-4585-a44e-f9c58e362b43.c000.snappy.parquet, range: 191727616-281354675, partition values: [empty row]
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-169998034-10.210.23.11-1507067630530:blk_83565156183_82548880660 file/path/to/data/part-00320-7915e327-3214-4585-a44e-f9c58e362b43.c000.snappy.parquet No live nodes contain current block Block locations: DatanodeInfoWithStorage[10.209.145.174:50010,DS-c7c0a172-5ffa-4f90-bfb5-717fb1e9ecf2,DISK] DatanodeInfoWithStorage[10.3.22.142:50010,DS-a1ba9ac9-dc92-4131-a2c2-9f7d03b97caf,DISK] DatanodeInfoWithStorage[10.209.146.156:50010,DS-71d8ae97-15d3-454e-a715-d9490e184989,DISK] Dead nodes:  DatanodeInfoWithStorage[10.209.146.156:50010,DS-71d8ae97-15d3-454e-a715-d9490e184989,DISK] DatanodeInfoWithStorage[10.209.145.174:50010,DS-c7c0a172-5ffa-4f90-bfb5-717fb1e9ecf2,DISK] DatanodeInfoWithStorage[10.3.22.142:50010,DS-a1ba9ac9-dc92-4131-a2c2-9f7d03b97caf,DISK]
{noformat}
Bug: Sporadic org.apache.spark.sql.connect.service.SparkConnectServiceE2ESuite failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Dec/24 15:40
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/12249143023/job/34169967121

[info] org.apache.spark.sql.connect.service.SparkConnectServiceE2ESuite *** ABORTED *** (1 second, 198 milliseconds)
[info]   java.lang.AssertionError: assertion failed
[info]   at scala.Predef$.assert(Predef.scala:264)
[info]   at org.apache.spark.sql.connect.service.SparkConnectSessionManager.invalidateAllSessions(SparkConnectSessionManager.scala:293)
[info]   at org.apache.spark.sql.connect.SparkConnectServerTest.clearAllExecutions(SparkConnectServerTest.scala:83)
[info]   at org.apache.spark.sql.connect.SparkConnectServerTest.clearAllExecutions$(SparkConnectServerTest.scala:80)
[info]   at org.apache.spark.sql.connect.service.SparkConnectServiceE2ESuite.clearAllExecutions(SparkConnectServiceE2ESuite.scala:27)
[info]   at org.apache.spark.sql.connect.SparkConnectServerTest.beforeEach(SparkConnectServerTest.scala:72)
[info]   at org.apache.spark.sql.connect.SparkConnectServerTest.beforeEach$(SparkConnectServerTest.scala:70)
[info]   at org.apache.spark.sql.connect.service.SparkConnectServiceE2ESuite.beforeEach(SparkConnectServiceE2ESuite.scala:27)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:233)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:334)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
Bug: Sporadic org.apache.spark.sql.connect.planner.SparkConnectServiceSuite failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Dec/24 15:38
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/12247020792/job/34164090771

https://github.com/apache/spark/actions/runs/12192688663/job/34013732292 

[info] - SPARK-43923: commands send events ((sql_command {
[info]   sql: "select 1"
[info] }
[info] ,Some(0))) *** FAILED *** (10 seconds, 27 milliseconds)
[info]   The code passed to eventually never returned normally. Attempted 667 times over 10.002926667 seconds. Last failure message: VerifyEvents.this.listener.executeHolder.isDefined was false No events have been posted in Timeout(Span(10, Seconds)). (SparkConnectServiceSuite.scala:878)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at org.scalatest.enablers.Retrying$$anon$4.tryTryAgain$2(Retrying.scala:219)
[info]   at org.scalatest.enablers.Retrying$$anon$4.retry(Retrying.scala:226)
[info]   at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:348)
[info]   at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:347)
[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:457)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite$VerifyEvents.executeHolder(SparkConnectServiceSuite.scala:878)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite$VerifyEvents.onCompleted(SparkConnectServiceSuite.scala:898)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$new$27(SparkConnectServiceSuite.scala:536)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$new$27$adapted(SparkConnectServiceSuite.scala:498)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$withCommandTest$4(SparkConnectServiceSuite.scala:833)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$withCommandTest$4$adapted(SparkConnectServiceSuite.scala:824)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$withEvents$1(SparkConnectServiceSuite.scala:853)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
Bug: Spark SQL:  `SELECT col1 AS col1` can cause erroneous col1 duplicity in plan nodes 
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/24 13:06
Affects Version/s: 3.5.0

{{ }}
We're writing a spark plugin that (among others) modifies plans, and are occasionally bit by this false duplicity. pyspark example:

 

{{from pyspark.sql.types import StructType, StructField, IntegerType, StringType}}
{{from pyspark.sql import Row}}

{{store_schema = StructType([}}
{{    StructField('s_store_sk', IntegerType(), True),}}
{{    StructField('s_state', StringType(), True),}}
{{    ])}}

{{store_data = [}}
{{    Row(s_store_sk=1, s_state="France"),}}
{{    Row(s_store_sk=2, s_state="France"),}}
{{    Row(s_store_sk=3, s_state="England"),}}
{{]}}

{{store_sales_schema = StructType([}}
{{    StructField('ss_net_profit', IntegerType(), True),}}
{{    StructField('ss_store_sk', IntegerType(), True),}}
{{    ])}}

{{store_sales_data = [}}
{{    Row(ss_store_sk=1, ss_net_profit=1000),}}
{{    Row(ss_store_sk=2, ss_net_profit=2000),}}
{{    Row(ss_store_sk=3, ss_net_profit=3000),}}
{{]}}

{{store = spark.createDataFrame(store_data, store_schema)}}

{{store.createOrReplaceTempView("store")}}

{{store_sales = spark.createDataFrame(store_sales_data, store_sales_schema)}}

{{store_sales.createOrReplaceTempView("store_sales")}}

{{spark.sql("""}}

{{    select s_state as s_state,     – <---- !}}
{{    rank() over ( partition by s_state order by sum(ss_net_profit) desc) as ranking}}
{{    from  store_sales, store}}
{{    where s_store_sk = ss_store_sk}}
{{    group by s_state}}{{    """).explain(mode="formatted")}}

 

Snippet from result:

{{...}}
{{ }}
{{(13) HashAggregate}}
{{Input [2]: s_state#1, sum#55L}}
{{Keys [1]: s_state#1}}
{{Functions [1]: sum(ss_net_profit#4)}}
{{Aggregate Attributes [1]: sum(ss_net_profit#4)#44L}}
{{Results [3]: s_state#1, s_state#1, sum(ss_net_profit#4)#44L AS _w1#45L   <--- !}}
{{ }}
{{(14) Sort}}
{{Input [3]: s_state#1, s_state#1, _w1#45L   <--- !}}
{{Arguments: s_state#1 ASC NULLS FIRST, _w1#45L DESC NULLS LAST, false, 0}}
{{ }}
{{(15) Window}}
{{Input [3]: s_state#1, s_state#1, _w1#45L   <--- !}}
{{Arguments: rank(_w1#45L) windowspecdefinition(s_state#1, _w1#45L DESC NULLS LAST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS ranking#41, s_state#1, _w1#45L DESC NULLS LAST}}
{{ }}
{{(16) Project}}
{{Output [2]: s_state#1, ranking#41}}
{{Input [4]: s_state#1, s_state#1, _w1#45L, ranking#41   <--- ! }}
{{...}}
Bug: Introduce an initial implementation of the single-pass Analyzer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/24 14:25
Affects Version/s: 4.0.0

This initial version should support simple SELECTs and FILTERs from the main relation types (json/csv/parquet/orc/hive) and basic expressions (binary logic and arithmetic).

Context: https://issues.apache.org/jira/browse/SPARK-49834
Bug: Spark Connect - Support Configurable Idle Timeout of Connect Server 
Status: Open | Priority: Major | Resolution: 
Created: 18/Dec/24 17:34
Affects Version/s: 3.5.3

This ticket proposes the creation of a configurable idle timeout window for the Spark Connect server to shutdown if no active sessions or requests are created/submitted/running. 

Currently, the Spark Connect server runs until manually terminated. For those who use Spark Connect on hosted map reduce clusters (e.g., Amazon EMR), this means the cluster is never marked as idle and, therefore, never shut down.

If this ticket were implemented, administrators would be able to provision clusters that automatically terminate if there is no active use of Spark Connect, thus saving resources.
Bug: UDFRegistration vs. functions.udf() differ in supported argument count (22 vs. 10)
Status: Open | Priority: Major | Resolution: 
Created: 19/Dec/24 23:23
Affects Version/s: 4.0.0

`spark.udf.register` and `functions.udf()` differ in the supported argument count for Java / Scala UDFs.

[UDFRegistration|https://github.com/apache/spark/blob/09d93ece2cf78ff265c305fcf79c5a4f4b5bd5a4/sql/api/src/main/scala/org/apache/spark/sql/api/UDFRegistration.scala#L110-L140] supports up to 22 parameters, but [udf()|https://github.com/apache/spark/blob/09d93ece2cf78ff265c305fcf79c5a4f4b5bd5a4/sql/api/src/main/scala/org/apache/spark/sql/functions.scala#L8543-L8586] only supports 10.

The impact of this is that it's hard for users to register UDFs marked as non-deterministic because they can't use the `udf()` syntax. 

I suspect that this is a simple accidental omission. I believe this is a straightforward, mechanical fix by re-running the quoted code-generation, so I'm filing this Jira in case someone with bandwidth wants to pick this up and fix it.
Bug: Transpose should show a proper error message when invalid index columns are specified
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/24 01:13
Affects Version/s: 4.0.0

{code:python}
>>> df = spark.range(10).transpose(sf.col("id") + 1)
Traceback (most recent call last):
...
py4j.protocol.Py4JJavaError: An error occurred while calling o40.transpose.
: org.apache.spark.SparkException: [INTERNAL_ERROR] Found the unresolved operator: 'UnresolvedTranspose [unresolvedalias((id#0L + cast(1 as bigint)))] SQLSTATE: XX000
...
{code}

It shouldn't show {{INTERNAL_ERROR}}.
Bug: connect createDataFrame schema logic bug
Status: Open | Priority: Major | Resolution: 
Created: 19/Dec/24 20:46
Affects Version/s: 3.5.3

if a schema ddl_string is passed in via createDataFrame
{code:python}
data = [{"id": 1, "value": 0}]
spark.createDataFrame(data, schema="id long, value double")
{code}
then it does not pass that through to spark connect, but instead *always* passes in the inferred schema as a json string.

a few notable issues: 
we can see the branch where the schema is passed in [here|https://github.com/apache/spark/blob/6add9c89855f9311d5e185774ddddcbf4323beee/python/pyspark/sql/connect/session.py#L467]

which eagerly calls the connect analyze method, but then the evaluated schema is only used if it's [pandas|https://github.com/apache/spark/blob/6add9c89855f9311d5e185774ddddcbf4323beee/python/pyspark/sql/connect/session.py#L523] or if its a [pyarrow table |https://github.com/apache/spark/blob/6add9c89855f9311d5e185774ddddcbf4323beee/python/pyspark/sql/connect/session.py#L604]

but never when you pass in [python literals|https://github.com/apache/spark/blob/6add9c89855f9311d5e185774ddddcbf4323beee/python/pyspark/sql/connect/session.py#L659]

for python literals it seems to always match on [this branch|https://github.com/apache/spark/blob/6add9c89855f9311d5e185774ddddcbf4323beee/python/pyspark/sql/connect/session.py#L709] which passes in the inferred schema.

it appears to do this because of the schema check
{code:python}
        # schema from spark.createDataFrame(data, schema="id long, value double") is a string
        if isinstance(schema, str):
            schema = self.client._analyze(  # type: ignore[assignment]
                method="ddl_parse", ddl_string=schema
            ).parsed

        # the schema from the client is StructType, so this always matches
        if isinstance(schema, (AtomicType, StructType)):
            _schema = schema
            if isinstance(schema, StructType):
                _num_cols = len(schema.fields)
            else:
                _num_cols = 1
        
        # but then down on line 708, we check _schema, which will always be Some when a string is passed in.
        if _schema is not None:
            local_relation = LocalRelation(_table, schema=_schema.json())
        else:
            local_relation = LocalRelation(_table)
{code}
so for summary
 - the connect.analyze method should not be eagerly called if the schema string is passed in, and its using python literals. It should only be evaluated when needed for pandas and pyarrow.
 - the schema string should be passed in to `LocalRelation` instead of the inferred json schema when provided.
Bug: Dead lock caused by SQLConf$ and SqlApiConf initialization
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 19/Dec/24 06:27
Affects Version/s: 3.5.0

{code:java}
"Executor 92 task launch worker for task 2910, task 69.0 in stage 7.0 (TID 2910)" #152 daemon prio=5 os_prio=0 cpu=616.25ms elapsed=258408.34s tid=0x00007f77d67cc330 nid=0x22c9e in Object.wait()  [0x00007f77755fb000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.spark.sql.internal.SQLConf$.<init>(SQLConf.scala:184)
	- waiting on the Class initialization monitor for org.apache.spark.sql.internal.SqlApiConf$
	at org.apache.spark.sql.internal.SQLConf$.<clinit>(SQLConf.scala) {code}
 

 
{code:java}
"Executor 92 task launch worker for task 5362, task 521.0 in stage 10.0 (TID 5362)" #123 daemon prio=5 os_prio=0 cpu=2443.78ms elapsed=258409.29s tid=0x00007f77d60ecad0 nid=0x22c7c in Object.wait()  [0x00007f777e591000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Class.forName0(java.base@17.0.6/Native Method)
	- waiting on the Class initialization monitor for org.apache.spark.sql.internal.SQLConf$
	at java.lang.Class.forName(java.base@17.0.6/Class.java:467)
	at org.apache.spark.util.SparkClassUtils.classForName(SparkClassUtils.scala:41)
	at org.apache.spark.util.SparkClassUtils.classForName$(SparkClassUtils.scala:36)
	at org.apache.spark.util.SparkClassUtils$.classForName(SparkClassUtils.scala:83)
	at org.apache.spark.sql.internal.SqlApiConf$.$anonfun$new$1(SqlApiConf.scala:73) {code}
 

For the first one:
{{at org.apache.spark.sql.internal.SQLConf$.<clinit>(SQLConf.scala)}}  {{- waiting on the Class initialization monitor for org.apache.spark.sql.internal.SqlApiConf$}}

{{For the second one.}} 
{{at org.apache.spark.sql.internal.SqlApiConf$.$anonfun$new$1(SqlApiConf.scala:73)}}  {{- waiting on the Class initialization monitor for org.apache.spark.sql.internal.SQLConf$}} 

 

 

See full thread dump in the attachment [^iceberg_compaction.txt]
Bug: Modify state eviction metrics in TransformWithState to represent physical deletions
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/24 04:18
Affects Version/s: 4.0.0

In SPARK-50302, we modified the indexing logic for TWS variables with TTL. As part of these changes, we made the `numRowsRemoved` parameter represent the number of logical state entries that were evicted as a result of TTL expiration. However, the `numRowsRemoved` parameter, by its name, should be about the number of underlying state store _rows_ that were removed. This is a physical metric, not a logical metric.

The logical metric that represents TTL expiration is `numValuesRemovedDueToTTLExpiry`, and that can be used for non-physical metrics. This fix should be made across all state variables—value, map, and list.
Bug: we should fix the decimal precision on function toInspector(dataType: DataType) in HiveInspectors class
Status: Open | Priority: Major | Resolution: 
Created: 18/Dec/24 08:58
Affects Version/s: 3.4.4

When we add some mask UDF on Decimal(18,10) type coloumn in hive-sdf moudle, the scale of the mask result value should be 10, but now is 18.

In JavaHiveDecimalObjectInspector class the default precision of Decimal is (38,18), so we need to change to the correct decimal precision on function toInspector(dataType: DataType) in HiveInspectors class.

Can somebody fix it?

 

STEP:

create table testtbl(c1 decimal(18,10), c2 double);

decimal type mask the result is :

!image-2024-12-19-08-54-19-895.png!
Bug: Expose metrics from BasePythonRunner
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/24 15:14
Affects Version/s: 3.5.3

BasePythonRunner only logs its metrics (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala#L521-L524)

It would be very helpful to expose them through operator-level metrics.
Bug: Improve HybridAnalyzer so it throws ExplicitlyUnsupportedResolverFeature in specific cases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/24 11:47
Affects Version/s: 4.0.0

Improve HybridAnalyzer so it throws ExplicitlyUnsupportedResolverFeature in specific cases (when
checkSupportedSinglePassFeatures is set to false).
Bug: Extend CTE logical plan classes with recursion parameter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/24 17:53
Affects Version/s: 4.0.0

PR adds new parameters to already existing classes UnresolvedWith, CTERelationRef and CTERelationDef to enable later implementation of recursive CTEs.
 * Additional parameters are added in pattern matching cases in other files of mentioned classes as well.
 * A large number of trivial changes in tests was required, due to new parameters introduced to the classes, which are also addressed in this PR.
 * More information for reviewers can be found here: [https://docs.google.com/document/d/1qcEJxqoXcr5cSt6HgIQjWQSqhfkSaVYkoDHsg5oxXp4/edit]
Bug: Refactor batch creation style in Optimizer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/24 14:32
Affects Version/s: 4.0.0

It is a pain to reorder batches and guard behavior of certain batches / sequences of batches by a flag. This was primarily due to ample usage of `{{{}::`{}}}, `{{{}:::`{}}}, and {{`:+`}} to juggle rules and batches around which imposed syntactic limitations. For example, see here: https://github.com/apache/spark/pull/49208/files#diff-11264d807efa58054cca2d220aae8fba644ee0f0f2a4722c46d52828394846efL209.

We'd like to refactor to use unified logic.
Bug: Structured streaming job failed due to checkpoint file does not exist
Status: Open | Priority: Major | Resolution: 
Created: 17/Dec/24 02:37
Affects Version/s: 3.2.0

Structured streaming job failed due to delta file does not exist

!Snipaste_2024-12-17_10-38-42.png!

 

we checked the access log for the missing file, and found that the file was never created. And we found that the correspond tmp file exists whose file name is [.506.delta.936315d9-b58b-4f18-b3ed-b413cd92646f.TID3368893.tmp|https://www.cosmos09.osdinfra.net/cosmos/searchDM/raw/AnaheimRawLogs/Anaheim/RealTimeCheckpoints/Execution0/20241215_application_1733980441401_12313/state/0/1721/.506.delta.936315d9-b58b-4f18-b3ed-b413cd92646f.TID3368893.tmp?property=info]
Bug: Inconsistent string truncation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/24 14:10
Affects Version/s: 4.0.0

Output of truncatedString is different for the same max fields:

{code:scala}
truncatedString(Seq(1, 2), "[", ", ", "]", 2)
{code}
outputs *"[1, 2]"*, but less elements for the same max fields:

{code:scala}
truncatedString(Seq(1, 2, 3), "[", ", ", "]", 2)
{code}
it outputs *"[1, ... 2 more fields]"* instead of *"[1, 2, ... 1 more fields]"*.
Bug: Store Union, Except and Intersect operator's output as lazy val
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/24 22:22
Affects Version/s: 4.0.0

Currently `Union`'s (same is for `Except` and `Intersect`) `output` is a `def`. This creates performance issues for queries with large number of stacked `UNION`s because of rules like `WidenSetOperationTypes` that traverse the logical plan and call `output` on each `Union` node. This has quadratic complexity: O(number_of_unions * (1 + 2 + 3  + ... + number_of_unions)).

The proposed solution is to use a `lazy val`.