Bug: Tests never succeed on pyspark 3.4.0 (work OK on pyspark 3.3.2)
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/23 20:32
Affects Version/s: 3.4.0

I have a project that uses pyspark. The tests have always run fine on pyspark versions prior to pyspark 3.4.0 but now fail on that version (which was released on 2023-04-13).

My project is configured to use the latest available version of pyspark:
{code:json}
dependencies = [
  "pyspark",
  "faker"
]
{code}
[https://github.com/jamiekt/jstark/blob/c1629cee4e4b8fb0b4471f6fc2941f1b0a99a4bf/pyproject.toml#L26-L29]

The tests are run using GitHub Actions. An example of the failing tests is at [https://github.com/jamiekt/jstark/actions/runs/4977164046], you can see there that the tests are run upon various combinations of OS & python version, they are all cancelled after running for over 5 hours.

If I [pin the version of pyspark to 3.3.2|https://github.com/jamiekt/jstark/commit/5fd7115d3719a7d6ef2547e8e35feb3ed76ee99f] then the tests all succeed in ~10 minutes, see [https://github.com/jamiekt/jstark/actions/runs/5061332947] for such a successful run.
----
This can be reproduced by cloning the repository and running only one test. The project uses hatch for managing environments and dependencies so you would need that installed ({{{}pipx install hatch{}}}/{{{}brew install hatch{}}}). I have reproduced the problem on python3.10.

Reproduce the problem by running these commands:
{code:bash}
# force use of python3.10
export HATCH_PYTHON=/path/to/python3.10
git clone https://github.com/jamiekt/jstark.git
cd jstark
# following command will create a virtualenv & install all dependencies, including pyspark 3.4.0
hatch run pytest -k test_basketweeks_by_product_and_customer
{code}
On my machine this never completes. I need to CTRL+C to crash out of it. I consider this to be equivalent behaviour to the tests that fail in the GitHub Actions pipeline after 6 hours.

Now let's checkout the branch which pins pyspark to 3.3.2 and run the same thing (the hatch environment will get rebuilt with pyspark 3.3.2)
{code:bash}
git checkout try-pyspark3-3-2
hatch run pytest -k test_basketweeks_by_product_and_customer
{code}
this time it succeeds in ~31seconds:
{code:bash}
➜  hatch run pytest -k test_basketweeks_by_product_and_customer
================================================================================================== test session starts ======================================================================================================================
platform darwin -- Python 3.10.10, pytest-7.3.1, pluggy-1.0.0
rootdir: /private/tmp/jstark
plugins: Faker-18.9.0, cov-4.0.0
collected 79 items / 78 deselected / 1 selected

tests/test_grocery_retailer_feature_generator.py .                                                                                                                                                                                                                       [100%]

================================================================= 1 passed, 78 deselected in 31.30s =================
{code}
That particular test constructs a very very complex pyspark dataframe which I suspect might be contributing to the problem, however the issue here is that it works on pyspark 3.3.2 but not on pyspark 3.4.0.