av9ash/GitBugs huggingface dataset:
Bug: Add scripts to start and stop Spark Connect server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 10:30
Affects Version/s: 3.4.0

Currently, there is no proper way to start and stop the Spark Connect server. Now it requires you to start it with, for example, a Spark shell:

{code}
# For development,
./bin/spark-shell \
   --jars `ls connector/connect/target/**/spark-connect*SNAPSHOT.jar` \
  --conf spark.plugins=org.apache.spark.sql.connect.SparkConnectPlugin
{code}

{code}
# For released Spark versions
./bin/spark-shell \
  --packages org.apache.spark:spark-connect_2.12:3.4.0 \
  --conf spark.plugins=org.apache.spark.sql.connect.SparkConnectPlugin
{code}

which is awkward.

We need some dedicated scripts for it.
Bug: Spark History Server fails to start on CentOS7 aarch64
Status: Open | Priority: Major | Resolution: 
Created: 07/Feb/23 08:54
Affects Version/s: 3.2.3

When I run `./sbin/start-history-server.sh`

I'll get the error below

!image-2023-02-07-16-54-43-593.png!

 

Although we already use org.openlabtesting.leveldbjni on aarch64, which can works on aarch64,

we still load org.fusesource.hawtjni.runtime.Library on wrong jar file

When we run `export SPARK_DAEMON_JAVA_OPTS=-verbose:class`, we can see the class is loaded from jline-2.14.6.jar where the correct class file is under leveldbjni-all-1.8.jar

 

Incorrect(now):

[Loaded org.fusesource.hawtjni.runtime.Library from file:/yourdir/spark/jars/jline-2.14.6.jar] 

Correct(expected):

[Loaded org.fusesource.hawtjni.runtime.Library from file:/yourdir/spark/jars/leveldbjni-all-1.8.jar]
Bug: Arrow string and binary vectors only support 1 GiB
Status: Open | Priority: Major | Resolution: 
Created: 04/Feb/23 20:30
Affects Version/s: 3.4.0

Since Arrow 10.0.0, BaseVariableWidthVector (the parent for string and binary vectors), only supports expanding up to 1 GiB through the safe interfaces, which Spark uses, instead of 2 GiB previously. This is due to [https://github.com/apache/arrow/pull/13815.] I added a comment in there but haven't got any responses yet, will make an issue in Arrow as well.

Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be {{{}2147483648{}}}, which is greater than {{Integer.MAX_VALUE}} which is {{{}2147483647{}}}, thus throwing a {{{}OversizedAllocationException{}}}.

See [https://github.com/apache/spark/pull/39572#issuecomment-1383195213] and the comment above for how I recreated to show this was now the case.
Bug: Spark SQL not use hive partition info
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/23 12:08
Affects Version/s: 3.2.1

I use spark3 to count partition num , like : 

table a is external parquet table, it have 3 partition columns (year ,month, day).

query sql : "select distinct month , day from a where year = '2022' "

i think spark can find hive metadata and use partition info, but it load all  "year = '2022'" partition data.

in spark2.4, it use TableLocalScanExec ,but spark3 use HiveTableRelation and scan hive parquet.
Bug: Skip Spark Connect tests if dependencies are not installed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 07:23
Affects Version/s: 3.4.0

{code}
arget/7411b1a1-5ebc-47a6-b3cb-c73dedc9a3c9/python3.9__pyspark.sql.tests.connect.test_parity_catalog__7iw4wnpb.log)
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/.../spark/python/pyspark/sql/tests/connect/test_connect_basic.py", line 29, in <module>
    from pyspark.sql.connect.client import Retrying
  File "/.../spark/python/pyspark/sql/connect/__init__.py", line 21, in <module>
    from pyspark.sql.connect.dataframe import DataFrame  # noqa: F401
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 50, in <module>
    import pyspark.sql.connect.plan as plan
  File "/.../spark/python/pyspark/sql/connect/plan.py", line 26, in <module>
    import pyspark.sql.connect.proto as proto
  File "/.../spark/python/pyspark/sql/connect/proto/__init__.py", line 18, in <module>
    from pyspark.sql.connect.proto.base_pb2_grpc import *
  File "/.../spark/python/pyspark/sql/connect/proto/base_pb2_grpc.py", line 19, in <module>
    import grpc
ModuleNotFoundError: No module named 'grpc'
{code}
Bug: Add Simple Scala UDFs to Scala/JVM Client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 13:26
Affects Version/s: 3.5.0

“Simple” here refers to UDFs that utilize no client-specific class files (e.g REPL-generated) and JARs. Essentially, a “simple” UDF may only reference in-built libraries and classes defined within the scope of the UDF.
Bug: DS V2 pushdown supports supports JDBC dialects compile `SortOrder` by themselves
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 08:49
Affects Version/s: 3.4.0

Currently, DS V2 pushdown framework compile the SortOrder in fixed format.
This is not flexible and friendly for some databases that do not support this syntax.
For example, the fixed format order by col asc nulls first is not supported by mssql server who doesn't support nulls first.
Bug: Job description in v2 FileWrites can have the wrong committer
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Feb/23 17:00
Affects Version/s: 3.3.1

There is a difference in behavior between v1 writes and v2 writes in the order of events happening when configuring the file writer and the committer.

v1:
 # writer.prepareWrite()
 # committer.setupJob()

v2:
 # committer.setupJob()
 # writer.prepareWrite()

 

This is because the `prepareWrite()` call (that is the one performing the call `
job.setOutputFormatClass(classOf[ParquetOutputFormat[Row]])`)
happens as part of the `createWriteJobDescription` which is `lazy val` in the `toBatch` call and therefore is evaluated after the `committer.setupJob` at the end of the `toBatch`

This causes issues when evaluating the committer as some elements might be missing, for example the aforementioned output format class not being set, causing the committer being set up as generic write instead of parquet write.

 

The fix is very simple and it is to make the `createJobDescription` call non-lazy
Bug: Add an option to use external storage to distribute JAR set in cluster mode on Kube
Status: Open | Priority: Minor | Resolution: 
Created: 06/Feb/23 19:47
Affects Version/s: 3.5.0

tl;dr – sometimes the driver can get overwhelmed serving the initial jar set. You'll see a lot of "Executor fetching spark://.../....jar" and then connection timed out.

 

On YARN the jars (in cluster mode) are cached in HDFS.
Bug: SPJ: encapsulate all SPJ related parameters in BatchScanExec
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Feb/23 20:39
Affects Version/s: 3.3.1

The list of SPJ parameters in {{BatchScanExec}} keeps growing, which is annoying since there are many places which do pattern-matching on {{BatchScanExec}} and they have to change accordingly. 

To make this less disruptive, we can introduce a struct for all the SPJ classes and use that as the parameter for {{BatchScanExec}}.
Bug: Remove obsolete/superfluous imports in spark-hadoop-cloud module
Status: Open | Priority: Minor | Resolution: 
Created: 23/Feb/23 10:17
Affects Version/s: 3.4.0

The explicit imports into hadoop-cloud are obsolete

* the hadoop-cloud-storage pom is a cut down export of the bindings to the various cloud stores in their hadoop-* modules
* it's been shipping since hadoop 2.10
* its grown to include cos and allyun support
* fairly well tested
* actually cuts removed support (hadoop-openstack) when withdrawn. Hadoop 3.3.5 has done this, leaving a stub jar there just to avoid breaking downstream builds like spark's current setup.

hadoop-cloud-storage *should* be all that's needed.

I know that the spark hadoop-2 profile still references the (long unsupported 2.7.x), but if you are using those releases then really you aren't going to talk to cloud infra
* no abfs connector
* s3n connector which won't authenticate with any of the aws regions launched in the past 5-8 years
* gcs connector won't work (its java11+; hadoop 3.2.x is minimum for java11 clients)
* none of the new chinese cloud services
* s3a connector very outdated.
* s3a connector using unshaded aws client which is unlikely to work with versions of jackson, httpclient written in the last 5 years, has trouble on java8 etc.

Proposed
* hadoop-2 profile to be the minimal hadoop-aws and hadoop-azure dependencies in the code today. cutting to the empty set would be better, but a bit more radical
* hadoop-3 profile to pull in hadoop-cloud-storage (excluding aws sdk as today), *and nothing else*

This will simplify everyone's life as there are fewer dependencies to reconcile. 

see also SPARK-39969 proposing making the hadoop-aws versions of the aws-sdk-bundle the normative one, as it is now newer than the spark-kinesis import and more broadly tested
Bug: Update DataTables to 1.13.2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Feb/23 12:49
Affects Version/s: 3.5.0

The 1.10.25 version of DataTables, that Spark uses, seems vulerable: https://security.snyk.io/package/npm/datatables.net/1.10.25.
It may or may not affect Spark, but updating to latest 1.13.2 seems doable.
Bug: Upgrade `cyclonedx-maven-plugin` to 2.7.6
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Feb/23 09:12
Affects Version/s: 3.5.0

[https://github.com/CycloneDX/cyclonedx-maven-plugin/releases/tag/cyclonedx-maven-plugin-2.7.4]

[https://github.com/CycloneDX/cyclonedx-maven-plugin/releases/tag/cyclonedx-maven-plugin-2.7.5]
Bug: Upgrade `maven-shade-plugin` to 3.4.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Feb/23 09:01
Affects Version/s: 3.4.0

* [https://github.com/apache/maven-shade-plugin/releases/tag/maven-shade-plugin-3.3.0]
 * https://github.com/apache/maven-shade-plugin/compare/maven-shade-plugin-3.3.0...maven-shade-plugin-3.4.1
Bug: RocksDB StateStore lock acquisition should happen after getting input iterator from inputRDD
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Feb/23 19:22
Affects Version/s: 3.5.0

The current behavior of the `{*}compute{*}` method in both `{*}StateStoreRDD{*}` and `{*}ReadStateStoreRDD{*}` is: we first get the state store instance and then get the input iterator for the inputRDD.

For RocksDB state store, the running task will acquire and hold the lock for this instance. The retried task or speculative task will fail to acquire the lock and eventually abort the job if there are some network issues. For example, When we shrink the executors, the alive one will try to fetch data from the killed ones because it doesn't know the target location (prefetched from the driver) is dead until it tries to fetch data. The query might be hanging for a long time as the executor will retry {{*spark.shuffle.io.maxRetries=3*}} times and for each retry wait for {{*spark.shuffle.io.connectionTimeout*}} (default value is 120s) before timeout. In total, the task could be hanging for about 6 minutes. And the retried or speculative tasks won't be able to acquire the lock in this period.

Making lock acquisition happen after retrieving the input iterator should be able to avoid this situation.
Bug: Error log improve ment for the lock acquisition of RocksDB state store instance
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Feb/23 18:47
Affects Version/s: 3.5.0

 
{code:java}
"23/02/23 23:57:44 INFO Executor: Running task 2.0 in stage 57.1 (TID 363)
"23/02/23 23:58:44 ERROR RocksDB StateStoreId(opId=0,partId=3,name=default): RocksDB instance could not be acquired by [ThreadId: Some(49), task: 3.0 in stage 57, TID 363] as it was not released by [ThreadId: Some(51), task: 3.1 in stage 57, TID 342] after 60002 ms.{code}
 


We are seeing those error messages for a testing query. The *taskId != partitionId* but we fail to be clear on this in the error log.

It's confusing when we see those logs: the second log entry seems to talk about `{*}task 3.0{*}` (it's actually partition 3 and retry attempt 0), but the `{*}TID 363{*}` is already occupied by `{*}task 2.0 in stage 57.1{*}`.

 

Also, it's unclear at which stage retry attempt, the lock is acquired (or fails to be acquired)
Bug: Optimize the packaging strategy of connect client module
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Feb/23 05:40
Affects Version/s: 3.5.0

# `perfmark-api` not shaded into `connect-client-jvm module.jar` and it is not the default dependency of spark, we can package `perfmark-api` to `connect-client-jvm module.jar` to avoid users' manual dependence on it
 # sbt-assembly result of `connect-client-jvm module` packed too much jars and not relocation them, such as hadoop, rocksdb, roaringbitmap and so on,  should simplify it refer to  the packaging results of maven
Bug: Clean up the third-party Java source code introduced by SPARK-27180
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Feb/23 06:47
Affects Version/s: 3.5.0

* resource-managers/yarn/src/test/java/org/apache/hadoop/net/ServerSocketUtil.java
 * resource-managers/yarn/src/test/java/org/eclipse/jetty/server/SessionManager.java
 * resource-managers/yarn/src/test/java/org/eclipse/jetty/server/session/SessionHandler.java
Bug: Remove 3.1 and Java 17 check from  filter condition of `testingVersions` in `HiveExternalCatalogVersionsSuite`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Feb/23 16:06
Affects Version/s: 3.5.0

Spark 3.1 already EOL and has been deleted from [https://dist.apache.org/repos/dist/release/spark,] we can simplifies the filter conditions of `testingVersions`, all version already support Java 17
Bug: Fix `native-image.propertie` in Scala Client
Status: Open | Priority: Minor | Resolution: 
Created: 15/Feb/23 15:38
Affects Version/s: 3.4.0

The content of `native-image.propertie` file is not correct. This file is used to create a native image using GraalVM see more info: https://docs.oracle.com/en/graalvm/enterprise/20/docs/reference-manual/native-image/BuildConfiguration/
https://www.graalvm.org/22.1/reference-manual/native-image/BuildConfiguration/

e.g.

The content in `META-INF/native-image/io.netty` should also relocated, just as in `grpc-netty-shaded`.

Now, the content of `META-INF/native-image/io.netty/netty-codec-http2/native-image.properties` is

```
Args = --initialize-at-build-time=io.netty \
       --initialize-at-run-time=io.netty.handler.codec.http2.Http2CodecUtil,io.netty.handler.codec.http2.Http2ClientUpgradeCodec,io.netty.handler.codec.http2.Http2ConnectionHandler,io.netty.handler.codec.http2.DefaultHttp2FrameWriter
```

but it should like
```
Args = --initialize-at-build-time=org.sparkproject.connect.client.io.netty \
       --initialize-at-run-time=org.sparkproject.connect.client.io.netty.handler.codec.http2.Http2CodecUtil,org.sparkproject.connect.client.io.netty.handler.codec.http2.Http2ClientUpgradeCodec,org.sparkproject.connect.client.io.netty.handler.codec.http2.Http2ConnectionHandler,org.sparkproject.connect.client.io.netty.handler.codec.http2.DefaultHttp2FrameWriter       
```
Other Transformer may need to be added

See more info in this discussion thread https://github.com/apache/spark/pull/39866#discussion_r1098833915
Bug: Remove unused blank line removal from CSVExprUtils
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Feb/23 11:40
Affects Version/s: 3.3.1

The non-multiline CSV read codepath contains references to removal of blank lines throughout. This is not necessary as blank lines are removed by the parser. Furthermore, it causes confusion, indicating that blank lines are removed at this point when instead they are already omitted from the data. The multiline code-path does not explicitly remove blank lines leading to what looks like disparity in behavior between the two.

The codepath for {{DataFrameReader.csv(dataset: Dataset[String])}} does need to explicitly skip lines, and this should be respected in {{CSVUtils}}.
Bug: Inconsistent data produced by `FlatMapCoGroupsInPandas`
Status: Open | Priority: Minor | Resolution: 
Created: 10/Feb/23 10:26
Affects Version/s: 3.3.0

We are seeing inconsistent data returned when using `FlatMapCoGroupsInPandas`. In the PySpark example from the comments, when we call `grouped_df.collect()` we get:

 

{{[Row(left_colms="Index(['cluster', 'event', 'abc'], dtype='object')", right_colms="Index(['cluster', 'event', 'def'], dtype='object')")] }}

 

When we call `grouped_df.show(5, truncate=False)` we get:

 

{{[Row(left_colms="Index(['cluster', 'abc'], dtype='object')", right_colms="Index(['cluster', 'event', 'def'], dtype='object')", xyz='1234')] }}

 

When we call `grouped_df_1.collect()` we get:

 

{{[Row(left_colms="Index(['cluster', 'abc'], dtype='object')", right_colms="Index(['cluster', 'event', 'def'], dtype='object')", xyz='1234')] }}
Bug: Close Live AppStore in the finally block for test cases
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/23 02:42
Affects Version/s: 3.5.0

AppStatusStore#createLiveStore will return RocksDB backend AppStatusStore when `
LIVE_UI_LOCAL_STORE_DIR` or `LIVE_UI_LOCAL_STORE_DIR`  is configured, should be closed it in finally block to release resources for test cases
Bug: Spark driver pod should not create executor pods when there is no driver service
Status: Open | Priority: Minor | Resolution: 
Created: 11/Feb/23 02:41
Affects Version/s: 3.3.1

Currently, the driver pod assumes the driver headless service exists when creating the executor pods. However, when this assumption doesn't hold, the driver would still spin up executor pods, and the executor pods would fail, and then the driver would try to create more pods, and so on. With this, the spark job doesn't make any progress, while it eats a lot of computational resource, and it won't reach to a terminal state until manual intervention (e.g. deleting the job or recreate the driver service).

 

This Jira Issue is to address this problem, by having the driver check the driver service before creating the executor pods.
Bug: Upgrade RoaringBitmap to 0.9.39
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Feb/23 03:20
Affects Version/s: 3.5.0

[https://github.com/RoaringBitmap/RoaringBitmap/releases/tag/0.9.39]
 * ForAllInRange Fixes Yet Again by [@larsk-db|https://github.com/larsk-db] in [#614|https://github.com/RoaringBitmap/RoaringBitmap/pull/614]
Bug: Upgrade some maven-plugins
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Feb/23 08:16
Affects Version/s: 3.5.0

[INFO]   maven-checkstyle-plugin ............................ 3.2.0 -> 3.2.1
[INFO]   maven-clean-plugin ................................. 3.1.0 -> 3.2.0
[INFO]   maven-dependency-plugin ............................ 3.3.0 -> 3.5.0
[INFO]   maven-enforcer-plugin ........................... 3.0.0-M2 -> 3.2.1
[INFO]   maven-source-plugin ................................ 3.1.0 -> 3.2.1
[INFO]   maven-surefire-plugin ........................ 3.0.0-M7 -> 3.0.0-M8
[INFO]   maven-jar-plugin ................................... 3.2.2 -> 3.3.0
Bug: Pass the comment option through to univocity if users set it explicitly in CSV dataSource
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Feb/23 10:55
Affects Version/s: 3.0.0

In PR [https://github.com/apache/spark/pull/29516], in order to fix some bugs, univocity-parsers used by CSV dataSource was upgraded from 2.8.3 to 2.9.0, it also involved a new feature of univocity-parsers that quoting values of the first column that start with the comment character. It made a breaking for users downstream that handing a whole row as input.
 
For codes:
{code:java}
Seq(("#abc", 1)).toDF.write.csv("/Users/guowei/comment_test") {code}
Before Spark 3.0，the content of output CSV files is shown as:
!image-2023-02-03-18-56-01-596.png!
After this change, the content is shown as:
!image-2023-02-03-18-56-10-083.png!

For users, they can't set comment option to '\u0000'  to keep the behavior as before because the new added `isCommentSet` check logic as follows:
{code:java}
val isCommentSet = this.comment != '\u0000'


def asWriterSettings: CsvWriterSettings = {
  // other code
  if (isCommentSet) {
    format.setComment(comment)
  }
  // other code
}
 {code}
It's better to pass the comment option through to univocity if users set it explicitly in CSV dataSource.

 

After this change, the behavior as flows:
|id|code|2.4 and before|3.0 and after|this update|remark|
|1|Seq("#abc", "\u0000def", "xyz").toDF()
.write.{color:#57d9a3}option("comment", "\u0000"){color}.csv(path)|#abc
*def*
xyz|{color:#4c9aff}"#abc"{color}
{color:#4c9aff}*def*{color}
{color:#4c9aff}xyz{color}|{color:#4c9aff}#abc{color}
{color:#4c9aff}*"def"*{color}
{color:#4c9aff}xyz{color}|{color:#4c9aff}this update has a little bit difference with 3.0{color}|
|2|Seq("#abc", "\u0000def", "xyz").toDF()
.write{color:#57d9a3}.option("comment", "#"){color}.csv(path)|#abc
*def*
xyz|"#abc"
*def*
xyz|"#abc"
*def*
xyz|the same|
|3|Seq("#abc", "\u0000def", "xyz").toDF()
.write.csv(path)|#abc
*def*
xyz|"#abc"
*def*
xyz|"#abc"
*def*
xyz|default behavior: the same|
|4|{_}Seq{_}("#abc", "\u0000def", "xyz").toDF().write.text(path)
spark.read.{color:#57d9a3}option("comment", "\u0000"){color}.csv(path)|#abc
xyz|{color:#4c9aff}#abc{color}
{color:#4c9aff}\u0000def{color}
{color:#4c9aff}xyz{color}|{color:#4c9aff}#abc{color}
{color:#4c9aff}xyz{color}|{color:#4c9aff}this update has a little bit difference with 3.0{color}|
|5|{_}Seq{_}("#abc", "\u0000def", "xyz").toDF().write.text(path)
spark.read.{color:#57d9a3}option("comment", "#"){color}.csv(path)|\u0000def
xyz|\u0000def
xyz|\u0000def
xyz|the same|
|6|{_}Seq{_}("#abc", "\u0000def", "xyz").toDF().write.text(path)
spark.read.csv(path)|#abc
xyz|#abc
\u0000def
xyz|#abc
\u0000def
xyz|default behavior: the same|
Bug: Allow drop column operation on non V2 tables
Status: In Progress | Priority: Minor | Resolution: 
Created: 02/Feb/23 11:29
Affects Version/s: 3.3.1

Drop column operations do not seem to be supported in non-V2 tables: https://github.com/apache/spark/blob/5d45a415f3a29898d92380380cfd82bfc7f579ea/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala#L93

We have not been able to find a clear rationale as to why. It would be nice if this feature can be supported for none V2 tables.
Bug: Transform LeftOuter join with IsNull filter on right side to Anti join
Status: In Progress | Priority: Minor | Resolution: 
Created: 06/Feb/23 14:39
Affects Version/s: 3.3.1

Optimize left outer join to anti join if exists IsNull filter on the right side.
For example,  query
{code:sql}
SELECT t1.*
FROM t1 LEFT JOIN t2
ON t1.id = t2.id
WHERE t2.id is null and t1.id = 1 and Rand() < 0.5
{code}
can be optimized to 
{code:sql}
SELECT t1.*
FROM t1 LEFT ANTI JOIN t2
ON t1.id = t2.id
WHERE t1.id = 1 and Rand() < 0.5
{code}
Bug: IntelliJ Build issue: value getArgument is not a member of org.mockito.invocation.InvocationOnMock
Status: Closed | Priority: Trivial | Resolution: Fixed
Created: 14/Feb/23 05:33
Affects Version/s: 3.2.4

When running the tests with IntelliJ, sometime the error pops out:

 

spark/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala:149:18
value getArgument is not a member of org.mockito.invocation.InvocationOnMock
invocation.getArgument[Identifier](0).name match {

 

It seems caused by some conflicts versioning of mockito in the IDE.