av9ash/GitBugs huggingface dataset:
Bug: Memory leak when using createDataFrame with persist
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/23 16:48
Affects Version/s: 3.5.0

When I create a dataset from pandas data frame and persisting it (DISK_ONLY), some "byte[]" objects (total size of imported data frame) will still remain in the driver's heap memory.

This is the sample code for reproducing it:
{code:python}
import pandas as pd
import gc

from pyspark.sql import SparkSession
from pyspark.storagelevel import StorageLevel

spark = SparkSession.builder \
    .config("spark.driver.memory", "4g") \
    .config("spark.executor.memory", "4g") \
    .config("spark.sql.execution.arrow.pyspark.enabled", "true") \
    .getOrCreate()

pdf = pd.read_pickle('tmp/input.pickle')
df = spark.createDataFrame(pdf)
df = df.persist(storageLevel=StorageLevel.DISK_ONLY)
df.count()

del pdf
del df
gc.collect()
spark.sparkContext._jvm.System.gc(){code}
After running this code, I will perform a manual GC in VisualVM, but the driver memory usage will remain at 550 MBs (at start it was about 50 MBs).

!CreateDataFrameWithoutUnpersist.png|width=467,height=349!

Then I tested with adding {{"df = df.unpersist()"}} after the {{"df.count()"}} line and everything was OK (Memory usage after performing manual GC was about 50 MBs).

!CreateDataFrameWithUnpersist.png|width=468,height=300!

Also, I tried with reading from parquet file (without adding unpersist line) with this code:
{code:python}
import gc

from pyspark.sql import SparkSession
from pyspark.storagelevel import StorageLevel

spark = SparkSession.builder \
    .config("spark.driver.memory", "4g") \
    .config("spark.executor.memory", "4g") \
    .config("spark.sql.execution.arrow.pyspark.enabled", "true") \
    .getOrCreate()

df = spark.read.parquet('tmp/input.parquet')
df = df.persist(storageLevel=StorageLevel.DISK_ONLY)
df.count()

del df
gc.collect()
spark.sparkContext._jvm.System.gc(){code}
Again everything was fine and memory usage was about 50 MBs after performing manual GC.

!ReadParquetWithoutUnpersist.png|width=473,height=302!
Bug: Replace explicit  `ArrayOps#toSeq` with `s.c.immutable.ArraySeq.unsafeWrapArray`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 14:09
Affects Version/s: 4.0.0

There is a behavioral difference between Scala 2.13 and 2.12 for explicit `ArrayOps.toSeq` calls, similar to the implicit conversion from `Array` to `Seq`.

In Scala 2.12, it returns a `mutable.WrappedArray`, which does not involve a collection copy.

```scala
Welcome to Scala 2.12.18 (OpenJDK 64-Bit Server VM, Java 17.0.9).
Type in expressions for evaluation. Or try :help.

scala> Array(1,2,3).toSeq
res0: Seq[Int] = WrappedArray(1, 2, 3)
```

However, in Scala 2.13, it returns an `immutable.ArraySeq` that with collection copy.

Since we have always used the non-collection copy behavior for this explicit conversion in the era of Scala 2.12, it is safe to assume that no collection copy is needed for Scala 2.13.
Bug: XML: keepInnerXmlAsRaw option
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 27/Nov/23 01:22
Affects Version/s: 4.0.0

Built-in XML data source gives related value and schema of the inner or nested elements. However, additional operations should be made by developers manually to convert unstructured data to structured, tabular format. If nested elements are kept in a format that is suitable with XML (for each level), we can convert them easily to a structured, tabular format with the existing methods that have already been developed (infer method of XmlInferSchema and parseColumn method of StaxXmlParser). Therefore there should be an option that affects StaxXmlParser and InferSchema classes to keep inner XML elements in their original or raw format.

https://github.com/apache/spark/pull/44022
Bug: Merge the parse rule of PercentileCont and PercentileDisc into functionCall
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/23 11:38
Affects Version/s: 4.0.0

Spark SQL parser have a special rule to parse [percentile_cont|percentile_disc](percentage) WITHIN GROUP (ORDER BY v).
We should merge this rule into the functionCall.
Bug: Fix `pyspark.pandas.tests.connect.computation.test_parity_apply_func` in Python 3.11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/23 23:08
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/6914662405/job/18816505612

{code}
======================================================================
ERROR [1.237s]: test_apply_batch_with_type (pyspark.pandas.tests.connect.computation.test_parity_apply_func.FrameParityApplyFunctionTests.test_apply_batch_with_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/pandas/tests/computation/test_apply_func.py", line 248, in test_apply_batch_with_type
    def identify3(x) -> ps.DataFrame[float, [int, List[int]]]:
                        ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13540, in __class_getitem__
    return create_tuple_for_frame_type(params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 721, in create_tuple_for_frame_type
    return Tuple[_to_type_holders(params)]
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 766, in _to_type_holders
    data_types = _new_type_holders(data_types, NameTypeHolder)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 832, in _new_type_holders
    raise TypeError(
TypeError: Type hints should be specified as one of:
  - DataFrame[type, type, ...]
  - DataFrame[name: type, name: type, ...]
  - DataFrame[dtypes instance]
  - DataFrame[zip(names, types)]
  - DataFrame[index_type, [type, ...]]
  - DataFrame[(index_name, index_type), [(name, type), ...]]
  - DataFrame[dtype instance, dtypes instance]
  - DataFrame[(index_name, index_type), zip(names, types)]
  - DataFrame[[index_type, ...], [type, ...]]
  - DataFrame[[(index_name, index_type), ...], [(name, type), ...]]
  - DataFrame[dtypes instance, dtypes instance]
  - DataFrame[zip(index_names, index_types), zip(names, types)]
However, got (<class 'int'>, typing.List[int]).

----------------------------------------------------------------------
Ran 10 tests in 78.247s

FAILED (errors=1)
{code}
Bug: Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/23 23:07
Affects Version/s: 3.5.0

https://github.com/apache/spark/actions/runs/6914662405/job/18812759697

{code}
======================================================================
ERROR [0.686s]: test_apply_batch_with_type (pyspark.pandas.tests.computation.test_apply_func.FrameApplyFunctionTests.test_apply_batch_with_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/pandas/tests/computation/test_apply_func.py", line 248, in test_apply_batch_with_type
    def identify3(x) -> ps.DataFrame[float, [int, List[int]]]:
                        ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13540, in __class_getitem__
    return create_tuple_for_frame_type(params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 721, in create_tuple_for_frame_type
    return Tuple[_to_type_holders(params)]
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 766, in _to_type_holders
    data_types = _new_type_holders(data_types, NameTypeHolder)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 832, in _new_type_holders
    raise TypeError(
TypeError: Type hints should be specified as one of:
  - DataFrame[type, type, ...]
  - DataFrame[name: type, name: type, ...]
  - DataFrame[dtypes instance]
  - DataFrame[zip(names, types)]
  - DataFrame[index_type, [type, ...]]
  - DataFrame[(index_name, index_type), [(name, type), ...]]
  - DataFrame[dtype instance, dtypes instance]
  - DataFrame[(index_name, index_type), zip(names, types)]
  - DataFrame[[index_type, ...], [type, ...]]
  - DataFrame[[(index_name, index_type), ...], [(name, type), ...]]
  - DataFrame[dtypes instance, dtypes instance]
  - DataFrame[zip(index_names, index_types), zip(names, types)]
However, got (<class 'int'>, typing.List[int]).

----------------------------------------------------------------------
Ran 10 tests in 34.327s

FAILED (errors=1)
{code}
Bug: pyspark.pandas read_excel implementation at version 3.4.1
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/23 20:19
Affects Version/s: 3.4.1

Hello, 

I would like to report an issue with pyspark.pandas implementation on read_excel function.

Microsoft Fabric spark environment 1.2 (runtime) uses pyspark 3.4.1 which potentially uses an older version of pandas on it's implementations of pyspark.pandas.

The function read_excel from pandas doesn't expect a parameter called "squeeze" however it's implemented as part of pyspark.pandas and the parameter "squeeze" is being passed to the pandas function.

 

!image-2023-11-28-13-20-40-275.png!

 

I've been digging into it for further investigation into pyspark 3.4.1 documentation

[https://spark.apache.org/docs/3.4.1/api/python/_modules/pyspark/pandas/namespace.html#read_excel|https://mcas-proxyweb.mcas.ms/certificate-checker?login=false&originalUrl=https%3A%2F%2Fspark.apache.org.mcas.ms%2Fdocs%2F3.4.1%2Fapi%2Fpython%2F_modules%2Fpyspark%2Fpandas%2Fnamespace.html%3FMcasTsid%3D20893%23read_excel&McasCSRF=92c0f0a0811f59386edd92fd5f3fcb0ac451ce363b3f2e01ed076f45e2b20500]

 

This is the point I found that "squeeze" parameter is being passed to pandas read_excel function which is not expected.

It seems like it was deprecated as part of pyspark 3.4.0 but still being used in the implementation.

 

!image-2023-11-28-13-20-51-291.png!

 

I believe this is an issue with pyspark implementation 3.4.1 not necessaily with fabric. However fabric uses this version as its 1.2 build.

 

I am able to work around that for now by download the excel from the one lake to the spark driver, loading that to the memory with pandas and then converting to a spark dataframe etc or I made it work downgrading the build

I downloaded the pyspark build 20230713 to my local, made the changes and re-compiled it and it worked locally. So it means that is related to the implementation and they would have to fix or I do a downgrade to older version like 3.3.3 or try the latest 3.5.0 which is not the case for fabric
Bug: hive table level setting hadoop.mapred.max.split.size
Status: Open | Priority: Major | Resolution: 
Created: 11/Nov/23 10:35
Affects Version/s: 3.5.0

In the scenario of hive table scan, by configuring the hadoop.mapred.max.split.size parameter, you can increase the parallelism of the scan hive table stage, thereby reducing the running time.


However, if a large table and a small table are in the same query, if only a separate hadoop.mapred.max.split.size parameter is configured, some stages will run a very large number of tasks, and some stages will The number of tasks running is very small. For runtime tasks, the hadoop.mapred.max.split.size parameter can be set separately for each hive table to ensure this balance.
Bug: If the hive table is a table, the outsourcing information will be displayed during ShowCreateTableCommand.
Status: Open | Priority: Major | Resolution: 
Created: 26/Nov/23 14:21
Affects Version/s: 3.5.0

For example:
CREATE EXTERNAL TABLE test_extaral_1 (a String);

When using SHOW CREATE TABLE test, if it is an external table, it is not displayed whether it is an external table.

spark-sql> show create table test_extaral_1;
createtab_stmt
CREATE TABLE `test`.`test_extaral_1` (
  `a` STRING)
USING orc
LOCATION '/test/test_extaral_1'

 

You can modify the display and see whether it is the appearance。

spark-sql> show create table test_extaral_1;
createtab_stmt

CREATE EXTERNAL TABLE `test`.`test_extaral_1` (
  `a` STRING)
USING orc
CREATE EXTERNAL TABLE `test`.`test_extaral_1` (
LOCATION '/test/test_extaral_1'
Bug: SparkContext add file should also copy file to local root path
Status: Open | Priority: Major | Resolution: 
Created: 21/Nov/23 11:31
Affects Version/s: 3.2.1

For below case, it failed with FileNotFoundException
{code:java}
add jar hdfs://path/search_hadoop_udf-1.0.0-SNAPSHOT.jar;
add file hdfs://path/feature_map.txt;

CREATE or replace TEMPORARY FUNCTION si_to_fn AS "com.shopee.deep.data_mart.udf.SlotIdToFeatName";

select si_to_fn(k, './feature_map.txt') as feat_name
from (    
    select 'slot_8116' as k    
    union all    
    select 'slot_2219' as k)
 A; {code}
It was called that user use valued-table then the task will running on driver, but driver didn't copy this file to the root path, then failed.
Bug: Upgrade Node.js to 18.X
Status: Open | Priority: Major | Resolution: 
Created: 20/Nov/23 17:39
Affects Version/s: 4.0.0

run docker build spark/dev/create-release/spark-rm
/Dockerfile and then I get this

Processing triggers for libc-bin (2.31-0ubuntu9.12) ...

================================================================================
================================================================================

                              DEPRECATION WARNING

  Node.js 12.x is no longer actively supported!

  You will not receive security or critical stability updates for this version.

  You should migrate to a supported version of Node.js as soon as possible.
  Use the installation script that corresponds to the version of Node.js you
  wish to install. e.g.

   * https://deb.nodesource.com/setup_16.x — Node.js 16 "Gallium"
   * https://deb.nodesource.com/setup_18.x — Node.js 18 LTS "Hydrogen" (recommended)
   * https://deb.nodesource.com/setup_19.x — Node.js 19 "Nineteen"
   * https://deb.nodesource.com/setup_20.x — Node.js 20 "Iron" (current)

  Please see https://github.com/nodejs/Release for details about which
  version may be appropriate for you.

  The NodeSource Node.js distributions repository contains
  information both about supported versions of Node.js and supported Linux
  distributions. To learn more about usage, see the repository:
    https://github.com/nodesource/distributions

================================================================================
================================================================================

Continuing in 20 seconds ...


================================================================================
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
================================================================================

                           SCRIPT DEPRECATION WARNING


  This script, located at https://deb.nodesource.com/setup_X, used to
  install Node.js is deprecated now and will eventually be made inactive.

  Please visit the NodeSource distributions Github and follow the
  instructions to migrate your repo.
  https://github.com/nodesource/distributions

  The NodeSource Node.js Linux distributions GitHub repository contains
  information about which versions of Node.js and which Linux distributions
  are supported and how to install it.
  https://github.com/nodesource/distributions


                          SCRIPT DEPRECATION WARNING

================================================================================
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
================================================================================

TO AVOID THIS WAIT MIGRATE THE SCRIPT
Continuing in 60 seconds (press Ctrl-C to abort) ...
Bug: Use Higher Order aggregate functions from SQL
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/23 09:25
Affects Version/s: 3.5.0

Higher order aggregate funtions are aggregation function which take a lambda function as a parameter.

An example for this from presto is the runction {{[reduce_agg|https://prestodb.io/docs/current/functions/aggregate.html#reduce_agg]}} which has the signature {{reduce_agg(inputValue T, initialState S, inputFunction(S, T, S), combineFunction(S, S, S))}} and it works like this:
{code:java}
SELECT id, reduce_agg(value, 0, (a, b) -> a + b, (a, b) -> a + b)
FROM (VALUES (1, 2), (1, 3), (1, 4), (2, 20), (2, 30), (2, 40)) AS t(id, value)
GROUP BY id;
-- (1, 9)
-- (2, 90)
{code}
In Spark you can today define, implement and use such a custom function from the scala API by implementing a case class which extends from {{TypedImperativeAggregate}} and add the {{HigherOrderFunction}} trait.

However if you try to use this function from the sql api, you get:
{code:java}
org.apache.spark.sql.AnalysisException: A lambda function should only be used in a higher order function. However, its class is org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression, which is not a higher order function.; line 2 pos 2
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:52)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.$anonfun$applyOrElse$155(Analyzer.scala:2142)
  at scala.Option.map(Option.scala:230)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.$anonfun$applyOrElse$154(Analyzer.scala:2135)
  at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:100)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.applyOrElse(Analyzer.scala:2143)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$153.applyOrElse(Analyzer.scala:2132)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
{code}
There is just a small thing missing in the {{Analyzer}} to get all of this working, we will provide a fix, unblocking higher order aggregate functions in spark sql.
Bug: Missing .jars when applying code to spark-connect
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Nov/23 11:32
Affects Version/s: 3.4.1

I've built spark with following maven code for our onprem hadoop cluster:
{code:bash}
./build/mvn -Pyarn -Pkubernetes -Dhadoop.version=3.1.2 -Pscala-2.12 -Phive -Phive-thriftserver -DskipTests clean package
{code}
 
So I start connect server like that:
{code:bash}
./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.4.1
{code}
 
When I'm trying to run any code after following code I always have an error from connect-server side:
{code:bash}
./bin/pyspark --remote "sc://localhost"
{code}
Error: 
{code:bash}
          /home/zeppelin/.ivy2/local/org.apache.spark/spark-connect_2.12/3.4.1/jars/spark-connect_2.12.jar

        ==== central: tried

          https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.4.1/spark-connect_2.12-3.4.1.pom

          -- artifact org.apache.spark#spark-connect_2.12;3.4.1!spark-connect_2.12.jar:

          https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.4.1/spark-connect_2.12-3.4.1.jar

        ==== spark-packages: tried

          https://repos.spark-packages.org/org/apache/spark/spark-connect_2.12/3.4.1/spark-connect_2.12-3.4.1.pom

          -- artifact org.apache.spark#spark-connect_2.12;3.4.1!spark-connect_2.12.jar:

          https://repos.spark-packages.org/org/apache/spark/spark-connect_2.12/3.4.1/spark-connect_2.12-3.4.1.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::          UNRESOLVED DEPENDENCIES         ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.apache.spark#spark-connect_2.12;3.4.1: not found

                ::::::::::::::::::::::::::::::::::::::::::::::
{code}
 

Where am I wrong? I thought it's a firewall issue what it's not cause I fixed to set http_proxy and https_proxy variables with my own credentials.
Bug: Janino compiler has new version that fix issue with compilation
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/23 10:35
Affects Version/s: 3.5.0

Janino released a new version (3.1.11) with a fix of a compilation of:
{code:java}
{
} while (fasle) {code}
[Link to github issue|https://github.com/janino-compiler/janino/issues/208]
Bug: Remove unnecessary TaskScheduler.killAllTaskAttempts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/23 08:28
Affects Version/s: 3.0.3

Spark has two functions to kill all tasks in a Stage:
* `cancelTasks`: Not only kill all the running tasks in all the stage attempts but also abort all the stage attempts
*  `killAllTaskAttempts`: Only kill all the running tasks in all the stage attemtps but won't abort the attempts.


However, there's no use case in Spark that a stage would launch new tasks after its all tasks get killed. So I think we can replace `killAllTaskAttempts` with `cancelTasks` directly.
Bug: -target is deprecated: Use -release instead
Status: Open | Priority: Major | Resolution: 
Created: 19/Nov/23 14:41
Affects Version/s: 4.0.0


 ./dev/make-distribution.sh --name custom-spark --pip -Pkubernetes

[INFO] --- scala:4.7.1:compile (scala-compile-first) @ spark-common-utils_2.13 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/bjorn/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.7.1-bin_2.13.12__61.0-1.7.1_20220712T022208.jar
[INFO] compiling 24 Scala sources and 31 Java sources to /home/bjorn/spark/common/utils/target/scala-2.13/classes ...
[WARNING] : -target is deprecated: Use -release instead to compile against the correct platform API.
Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation
[WARNING] one warning found
[WARNING] Unexpected javac output: warning: No processor claimed any of these annotations: /org.apache.spark.annotation.Evolving,/org.apache.spark.annotation.Private
1 warning.
Bug: Increase Test Coverage
Status: Open | Priority: Major | Resolution: 
Created: 21/Nov/23 23:51
Affects Version/s: 4.0.0

Our current PySpark test coverage stands at approximately 80% as reported on [Codecov, |https://app.codecov.io/gh/apache/spark]which shows a recent decrease in test coverage. It is essential that we make efforts to enhance our test coverage going forward.

Our objective is to significantly increase our PySpark test coverage to reach >{*}90%{*}.
Bug: Move ArtifactManager from Spark Connect into SparkSession (sql/core)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/23 15:47
Affects Version/s: 4.0.0

The `ArtifactManager` that currently lies in the connect package can be moved into the wider sql/core package (e.g SparkSession) to expand the scope. This is possible because the `ArtifactManager` is tied solely to the `SparkSession#sessionUUID` and hence can be cleanly detached from Spark Connect and be made generally available.
Bug: Refactor SparkConnectSessionManager to not use guava cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 18:26
Affects Version/s: 4.0.0

Guava cache gives limited control over session eviction. For example, there can't be more complex policies of session eviction. Refactor it to be more similar to SparkConnectExecutionManager.