av9ash/GitBugs huggingface dataset:
Bug: Support regex-based MetricFilter in StatsdSink
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/23 15:45
Affects Version/s: 4.0.0

Spark statsd metrics sink in the current state does not support metrics filtering option. 

Though this option is available in the reporters it is not exposed in the statsD sink. An example of this can be looked at [https://github.com/apache/spark/blob/be9ffb37585fe421705ceaa52fe49b89c50703a3/core/src/main/scala/org/apache/spark/metrics/sink/GraphiteSink.scala#L76]

 

This is a critical option to have when teams does not want all the metrics that are exposed by spark in the metrics monitoring platforms and switch to detailed metrics as and when needed.
Bug: Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Aug/23 08:22
Affects Version/s: 3.1.1

_SQL_ like below:

select tmp.* 
 from
 (select
        device_id, ads_id, 
        from_json(regexp_replace(device_personas, '(?<=(\\\\\\{|,))"device_', '"user_device_'), ${device_schema}) as tmp
        from input )

${device_schema} includes more than 100 fields.

if Rule: OptimizeCsvJsonExprs  been applied, the expression, regexp_replace, will be invoked many times, that costs so much time.
Bug: Improve HeartbeatReceiver config validation error message
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Aug/23 21:31
Affects Version/s: 4.0.0

{code}
$ bin/spark-shell -c spark.network.timeout=30s
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/08 14:38:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/08/08 14:38:19 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: spark.network.timeoutInterval should be less than or equal to spark.storage.blockManagerHeartbeatTimeoutMs.
{code}
Bug: INSET hash hset set to None when plan exported into JSON
Status: Open | Priority: Minor | Resolution: 
Created: 08/Aug/23 18:45
Affects Version/s: 3.4.1

I am exporting optimized plans using `_jdf.queryExecution().optimizedPlan().toJSON()`. I noticed that when the plan contains a `INSET` operator the `hset` attribute is None (instead of containing the set elements).

 

When printing directly `_jdf.queryExecution().optimizedPlan()` the `INSET` operator has all the elements so I guess that the problem is with the `toJSON` method.
Bug: Logging level isn't passed to RocksDB state store provider correctly
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/23 23:57
Affects Version/s: 3.4.1

We pass log4j's log level to RocksDB so that RocksDB debug log can go to log4j. However, we pass in log level after we create the logger. However, the way it is set isn't effective. This has two impacts: (1) setting DEBUG level don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR level does prevent INFO level logging, but RocksDB still makes JNI calls to Scala, which is an unnecessary overhead.
Bug: Clean up the usage of org.apache.commons.lang3.RandomUtils
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Aug/23 04:08
Affects Version/s: 4.0.0

{code:java}
/**
 * Utility library that supplements the standard {@link Random} class.
 *
 * <p>Caveat: Instances of {@link Random} are not cryptographically secure.</p>
 *
 * <p>Please note that the Apache Commons project provides a component
 * dedicated to pseudo-random number generation, namely
 * <a href="https://commons.apache.org/proper/commons-rng/">Commons RNG</a>, that may be
 * a better choice for applications with more stringent requirements
 * (performance and/or correctness).</p>
 *
 * @deprecated Use Apache Commons RNG's optimized <a href="https://commons.apache.org/proper/commons-rng/commons-rng-client-api/apidocs/org/apache/commons/rng/UniformRandomProvider.html">UniformRandomProvider</a>
 * @since 3.3
 */
@Deprecated
public class RandomUtils { {code}
In {{commons-lang3}} 3.13.0, {{RandomUtils}} has been marked as {{@Deprecated}}
Bug: Cannot derive ExpressionEncoder when using types tagged by a trait
Status: Open | Priority: Minor | Resolution: 
Created: 07/Aug/23 09:18
Affects Version/s: 3.1.3

Steps to reproduce:
{code:java}
trait Tag
case class Foo(x: Int)
case class Bar(x: Foo with Tag)
ExpressionEncoder.apply[Bar](){code}
the ExpressionEncoder throws the following error (this was for 3.1.2):
{code:java}
scala.MatchError: Foo with Tag (of class scala.reflect.internal.Types$RefinedType0)
  at org.apache.spark.sql.catalyst.ScalaReflection.getConstructorParameters(ScalaReflection.scala:931)
  at org.apache.spark.sql.catalyst.ScalaReflection.getConstructorParameters$(ScalaReflection.scala:928)
  at org.apache.spark.sql.catalyst.ScalaReflection$.getConstructorParameters(ScalaReflection.scala:49) {code}
The bug is [on this line (on master)|https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L461].
{code:java}
val TypeRef(_, _, actualTypeArgs) = dealiasedTpe {code}
which is an incomplete pattern match. The pattern match needs be extended to include matching on RefinedType as well.
Bug: Fix git ignore rules related to Antlr
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/23 08:29
Affects Version/s: 3.5.0

run `git status` after SPARK-44475 merged

 
{code:java}
sql/api/gen/
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/gen/ {code}
The rules should be modified in the .gitignore file to ignore them.
Bug: org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#resultSetToRows produces wrong data
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 02/Aug/23 00:17
Affects Version/s: 2.3.2

When the resultSet exists a timestmp column and it's value is null, but column define is not null. In the row it generates, this column will use the value of the same column in the previous row.  

 

In mysql, if a datetime column is defined, meanwhile it is not null. When a value is '0000-00-00 00:00:00', mysql provided a property of zeroDateTimeBehavior, it will return null.  

table define:
CREATE TABLE `test_timestamp` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键id',
`unbind_time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00' ,
PRIMARY KEY (`id`)
)
example:

the value of resultSet

1, 2023-01-01 12:00:00

2, null

 

the value of row

1, 2023-01-01 12:00:00

2, 2023-01-01 12:00:00
Bug: Adjust Pull Request Template to incorporate the ASF Generative Tooling Guidance recommendations
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 12/Aug/23 10:18
Affects Version/s: 3.3.2

Recently releases [ASF Generative Tooling Guidance|https://www.apache.org/legal/generative-tooling.html] recommends keeping track of the generative AI tools used to author patches

??When providing contributions authored using generative AI tooling, a recommended practice is for contributors to indicate the tooling used to create the contribution. This should be included as a token in the source control commit message, for example including the phrase “Generated-by: ”. This allows for future release tooling to be considered that pulls this content into a machine parsable Tooling-Provenance file.??

We should adjust PR template accordingly.
Bug: Not waiting for configmap before starting driver
Status: Open | Priority: Major | Resolution: 
Created: 31/Jan/22 18:12
Affects Version/s: 3.2.0

*The problem*

When you spark-submit to kubernetes in cluster-mode:
 # Kubernetes creates the driver
 # Kubernetes creates a configmap that the driver depends on

This is a race condition. If the configmap is not created quickly enough, then the driver will fail to start up properly.

See [this stackoverflow post|https://stackoverflow.com/a/58508313] for an alternate description of this problem.

 

*To Reproduce*

 # Download spark 3.2.0 or 3.2.1 from [https://spark.apache.org/downloads.html]
 # Create an image with 
{code:java}
bin/docker-image-tool.sh{code}
 # Spark submit one of the examples to some kubernetes instance
 # Observe the race condition
Bug: Analyzer can fail when join left side and right side are the same view
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 10:49
Affects Version/s: 3.2.0

See the attachment for details, including SQL and the exception information.
 * sql1, there is a normal filter (LO_SUPPKEY > 10) in the right side subquery, Analyzer works as expected;
 * sql2, there is a HAVING filter(HAVING COUNT(DISTINCT LO_SUPPKEY) > 1) in the right side subquery, Analyzer failed with "Resolved attribute(s) LO_SUPPKEY#337 missing ...".

      From the debug info, the problem seems to be occurred after the rule DeduplicateRelations is applied.
Bug: Migrate onto error classes
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jan/22 16:28
Affects Version/s: 3.3.0

The PR https://github.com/apache/spark/pull/32850 introduced error classes as a part of the error messages framework (https://issues.apache.org/jira/browse/SPARK-33539). Need to migrate all exceptions from QueryExecutionErrors, QueryCompilationErrors and QueryParsingErrors on the error classes using instances of SparkThrowable, and carefully test every error class by writing tests in dedicated test suites:
*  QueryExecutionErrorsSuite for the errors that are occurred during query execution
* QueryCompilationErrorsSuite ... query compilation or eagerly executing commands
* QueryParsingErrorsSuite ... parsing errors

Here is an example https://github.com/apache/spark/pull/35157 of how an existing Java exception can be replaced, and testing of related error classes.At the end, we should migrate all exceptions from the files Query.*Errors.scala and cover all error classes from the error-classes.json file by tests.
Bug: An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/22 16:03
Affects Version/s: 3.0.0

Doing an outer-join using {{joinWith}} on {{{}DataFrame{}}}s used to return missing values as {{null}} in Spark 2.4.8, but returns them as {{Rows}} with {{null}} values in Spark 3+.

The issue can be reproduced with [the following test|https://github.com/cdegroc/spark/commit/79f4d6a1ec6c69b10b72dbc8f92ab6490d5ef5e5] that succeeds on Spark 2.4.8 but fails starting from Spark 3.0.0.

The problem only arises when working with DataFrames: Datasets of case classes work as expected as demonstrated by [this other test|https://github.com/apache/spark/blob/v3.0.0/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala#L1200-L1223].

I couldn't find an explanation for this change in the Migration guide so I'm assuming this is a bug.

A {{git bisect}} pointed me to [that commit|https://github.com/apache/spark/commit/cd92f25be5a221e0d4618925f7bc9dfd3bb8cb59].

Reverting the commit solves the problem.

A similar solution,  but without reverting, is shown [here|https://github.com/cdegroc/spark/commit/684c675bf070876a475a9b225f6c2f92edce4c8a].

Happy to help if you think of another approach / can provide some guidance.
Bug: Use error classes in the compilation errors of properties
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 17:24
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* cannotReadCorruptedTablePropertyError
* cannotCreateJDBCNamespaceWithPropertyError
* cannotSetJDBCNamespaceWithPropertyError
* cannotUnsetJDBCNamespaceWithPropertyError
* alterTableSerDePropertiesNotSupportedForV2TablesError
* unsetNonExistentPropertyError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Aggregation with Watermark in AppendMode is holding data beyong water mark boundary.
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 31/Jan/22 16:11
Affects Version/s: 3.2.0

 I am struggling with a unique issue. I am not sure if my understanding is wrong or this is a bug with spark.
 
 #  I am reading a stream from events hub/kafka ( Extract)
 #  Pivoting and Aggregating the above dataframe ( Transformation). This is a WATERMARKED aggregation.
 #  writing the aggregation to Console/Delta table in APPEND  mode with a Trigger . 

However, the most recently published message to event hub is not writing to console/delta even after falling out of the watermark time. 
 
 My understanding is the event should be inserted to  the Delta table after Eventtime+Watermark.
 

Moreover, all the events in the memory stored must be flushed out to the sink irrespective of the watermark before stopping to mark a graceful shutdown .

 

Please advise.
Bug: Throw Spark exceptions from AES functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/22 19:24
Affects Version/s: 3.3.0

Currently, Spark SQL can throw Java exceptions from the aes_encrypt()/aes_decrypt() functions, for instance:

{code:java}
java.lang.RuntimeException: javax.crypto.AEADBadTagException: Tag mismatch!
	at org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.aesInternal(ExpressionImplUtils.java:93)
	at org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.aesDecrypt(ExpressionImplUtils.java:43)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.project_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:354)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
	at com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:620)
	at com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
	at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
	at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
	at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
	at javax.crypto.Cipher.doFinal(Cipher.java:2226)
	at org.apache.spark.sql.catalyst.expressions.ExpressionImplUtils.aesInternal(ExpressionImplUtils.java:87)
	... 19 more
{code}

That might confuse non-Scala/Java users. Need to wrap such kind of exception by Spark's exception.
Bug: Extend METADATA column to support row indices for file based data sources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/22 16:13
Affects Version/s: 3.3.0

Spark recently added hidden metadata column support for File based datasources as part of  SPARK-37273.

We should extend it to support ROW_INDEX/ROW_POSITION also.

 

Meaning of  ROW_POSITION:

ROW_INDEX/ROW_POSITION is basically an index of a row within a file. E.g. 5th row in the file will have ROW_INDEX 5.

 

Use cases: 

Row Indexes can be used in a variety of ways. A (fileName, rowIndex) tuple uniquely identifies row in a table. This information can be used to mark rows e.g. this can be used by indexer etc.
Bug: Use error classes in the parsing errors of intervals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 16:45
Affects Version/s: 3.3.0

Modify the following methods in QueryParsingErrors:
 * moreThanOneFromToUnitInIntervalLiteralError
 * invalidIntervalLiteralError
 * invalidIntervalFormError
 * invalidFromToUnitValueError
 * fromToIntervalUnsupportedError
 * mixedIntervalUnitsError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: ExecutorMonitor.timedOutExecutors should be deterministic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/22 08:13
Affects Version/s: 3.2.0

Since the AS-IS timedOutExecutors returns the result indeterministic, it kills the executors in a random order at Dynamic Allocation setting.

spark/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala

{code}
 private val executors = new ConcurrentHashMap[String, Tracker]() 
...
 timedOutExecs = executors.asScala 
{code}

This random behavior not only makes the users confusing but also causes a K8s decommission tests flaky like the following case in Java 17 on Apple Silicon environment. The K8s test expects the decommission of executor 1 while the executor 2 is chosen at this time.
{code}
22/01/25 06:11:16 DEBUG ExecutorMonitor: Executors 1,2 do not have active shuffle data after job 0 finished.
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: max needed for rpId: 0 numpending: 0, tasksperexecutor: 1
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: No change in number of executors
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: Request to remove executorIds: (2,0), (1,0)
22/01/25 06:11:16 DEBUG ExecutorAllocationManager: Not removing idle executor 1 because there are only 1 executor(s) left (minimum number of executor limit 1)
22/01/25 06:11:16 INFO KubernetesClusterSchedulerBackend: Decommission executors: 2
{code}
Bug: log4j update to 2.17.1 in spark-core 3.2
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 12/Jan/22 15:16
Affects Version/s: 3.2.0

We are using spark-core jar file, as below

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.2.0"

as per maven repository it usgaes log4j older version, which need to be updated latest(2.17.1) to fix security vulenrability, please help us, how we can get update version spark-core, which usages latest updated log4j.

Thanks
Bug: Apply and evaluate expressions row-wise in a Spark DataFrame
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/22 14:48
Affects Version/s: 3.2.0

This functionality would serve very specific use cases.

Consider a DataFrame with a column of SQL expressions encoded as strings. Individually it's possible to evaluate each string and obtain the corresponding result with PySpark's expr function. However it is not possible to apply the expr function row-wise to a dataframe (UDF or map), and evaluate all expressions efficiently.
{code:java}
id  |  sql_expression
--------------------------
1   |  abs(-1) + 12
2   |  decode(1,2,3,4) - 1
3   |  30 * 20 - 5

df = df.withColumn('sql_eval', f.expr_row('sql_expression')) {code}
Bug: Disable mapreduce.fileoutputcommitter.algorithm.version=2 by default
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 18/Jan/22 02:30
Affects Version/s: 3.2.0

The hadoop MR v2 commit algorithm had a correctness issue described by SPARK-33019, and changed spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=1 by default. But some spark users like me ware unaware of this correctness issue before and had used v2 commit algorithm in spark 2.x for performance purposes. And after upgrade to spark 3.x, we encountered this correctness issue in production environment, caused a very serious failure.The trigger probability of this issue was higher in new version spark 3.x, and I didn't delve into the specific reasons. So I propose we should better disable spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 by default, if users using v2 commit algorithm, then fail the job and warn users this correctness issue. Or users can choose to force the v2 usage through a new configuration.
Bug: Limit push down for parquet datasource v2
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jan/22 11:43
Affects Version/s: 3.4.0

Based on SPARK-37020, we can support limit push down to parquet datasource v2 reader. It can stop scanning parquet early, and reduce network and disk IO.

Current limit parse status for parquet
{code:java}
== Parsed Logical Plan ==
GlobalLimit 10
+- LocalLimit 10
   +- RelationV2[a#0, b#1] parquet file:/datasources.db/test_push_down

== Analyzed Logical Plan ==
a: int, b: int
GlobalLimit 10
+- LocalLimit 10
   +- RelationV2[a#0, b#1] parquet file:/datasources.db/test_push_down

== Optimized Logical Plan ==
GlobalLimit 10
+- LocalLimit 10
   +- RelationV2[a#0, b#1] parquet file:/datasources.db/test_push_down

== Physical Plan ==
CollectLimit 10
+- *(1) ColumnarToRow
   +- BatchScan[a#0, b#1] ParquetScan DataFilters: [], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/datasources.db/test_push_down/par..., PartitionFilters: [], PushedAggregation: [], PushedFilters: [], PushedGroupBy: [], ReadSchema: struct<a:int,b:int>, PushedFilters: [], PushedAggregation: [], PushedGroupBy: [] RuntimeFilters: [] {code}
Bug: Fix DataFrame select subset with duplicated columns
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jan/22 09:36
Affects Version/s: 3.3.0

pandas
{code:java}
>>> pdf
   a
0  1
1  2
2  3
3  4
>>> pdf[['a', 'a']]
   a  a
0  1  1
1  2  2
2  3  3
3  4  4 {code}
pandas on spark
{code:java}
>>> psdf
   a
0  1
1  2
2  3
3  4
>>> psdf[['a', 'a']]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/u02/spark/python/pyspark/pandas/frame.py", line 12077, in __repr__
    pdf = self._get_or_create_repr_pandas_cache(max_display_count)
  File "/u02/spark/python/pyspark/pandas/frame.py", line 12068, in _get_or_create_repr_pandas_cache
    self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
  File "/u02/spark/python/pyspark/pandas/frame.py", line 12063, in _to_internal_pandas
    return self._internal.to_pandas_frame
  File "/u02/spark/python/pyspark/pandas/utils.py", line 576, in wrapped_lazy_property
    setattr(self, attr_name, fn(self))
  File "/u02/spark/python/pyspark/pandas/internal.py", line 1055, in to_pandas_frame
    return InternalFrame.restore_index(pdf, **self.arguments_for_restore_index)
  File "/u02/spark/python/pyspark/pandas/internal.py", line 1156, in restore_index
    pdf.columns = pd.Index(
  File "/u02/venv3.9-2/lib/python3.9/site-packages/pandas/core/generic.py", line 5500, in __setattr__
    return object.__setattr__(self, name, value)
  File "pandas/_libs/properties.pyx", line 70, in pandas._libs.properties.AxisProperty.__set__
  File "/u02/venv3.9-2/lib/python3.9/site-packages/pandas/core/generic.py", line 766, in _set_axis
    self._mgr.set_axis(axis, labels)
  File "/u02/venv3.9-2/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 216, in set_axis
    self._validate_set_axis(axis, new_labels)
  File "/u02/venv3.9-2/lib/python3.9/site-packages/pandas/core/internals/base.py", line 57, in _validate_set_axis
    raise ValueError(
ValueError: Length mismatch: Expected axis has 4 elements, new values have 2 elements {code}