Bug: Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement
Status: Open | Priority: Major | Resolution: 
Created: 16/Apr/23 10:25
Affects Version/s: 3.4.0

After upgrading to Pyspark version 3.4.0 from version 3.3.2, workflows using the "pivot" function fails with the following error:

 
`File "/usr/local/lib/python3.8/site-packages/pyspark/sql/group.py", line 512, in pivot
[847|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L847] jgd = self._jgd.pivot(pivot_col, values)
[848|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L848] File "/usr/local/lib/python3.8/site-packages/py4j/java_gateway.py", line 1322, in __call__
[849|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L849] return_value = get_return_value(
[850|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L850] File "/usr/local/lib/python3.8/site-packages/pyspark/errors/exceptions/captured.py", line 169, in deco
[851|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L851] return f(*a, **kw)
[852|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L852] File "/usr/local/lib/python3.8/site-packages/py4j/protocol.py", line 330, in get_return_value
[853|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L853] raise Py4JError(
[854|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L854]py4j.protocol.Py4JError: An error occurred while calling o999.pivot. Trace:
[855|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L855]py4j.Py4JException: Method pivot([class java.lang.String, class [Ljava.lang.String;]) does not exist
[856|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L856] at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:321)
[857|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L857] at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:329)
[858|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L858] at py4j.Gateway.invoke(Gateway.java:274)
[859|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L859] at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
[860|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L860] at py4j.commands.CallCommand.execute(CallCommand.java:79)
[861|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L861] at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
[862|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L862] at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
[863|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L863] at java.base/java.lang.Thread.run(Thread.java:829)`
 
My workflow calls the "pivot" method using the "values" argument.
I didn't have the chance to try calling pivot without it to see if the error occurs also when omitting the "values" argument.