Bug: Pyspark throws AnalysisException with incorrect error message when using .grouping() or .groupingId() (AnalysisException: grouping() can only be used with GroupingSets/Cube/Rollup;)
Status: Open | Priority: Minor | Resolution: 
Created: 21/Apr/22 14:07
Affects Version/s: 3.1.2

h1. In a nutshell

Pyspark emits an incorrect error message when committing a type error with the results of the {{grouping()}} function.
h1. Code to reproduce

{{print(spark.version) # My environment, Azure DataBricks, defines spark automatically.}}
{{from pyspark.sql import functions as f}}
{{{}from pyspark.sql import types as t{}}}{{{}l = [{}}}
{{  ('a',),}}
{{  ('b',),}}
{{]}}
{{s = t.StructType([}}
{{  t.StructField('col1', t.StringType())}}
{{])}}
{{df = spark.createDataFrame(l, s)}}
{{{}df.display(){}}}{{{}( # This expression raises an AnalysisException(){}}}
{{  df}}
{{  .cube(f.col('col1'))}}
{{  .agg(f.grouping('col1') & f.lit(True))}}
{{  .collect()}}
{{)}}
h1. Expected results

The code produces an {{AnalysisException()}} with error message along the lines of:
{{AnalysisException: cannot resolve '(GROUPING(`col1`) AND true)' due to data type mismatch: differing types in '(GROUPING(`col1`) AND true)' (int and boolean).;}}
h1. Actual results

The code throws an {{AnalysisException()}} with error message
{{AnalysisException: grouping() can only be used with GroupingSets/Cube/Rollup;}}

Python provides the following traceback:
{{---------------------------------------------------------------------------}}
{{AnalysisException                         Traceback (most recent call last)}}
{{<command-2283735107422632> in <module>}}
{{     15 }}
{{     16 ( # This expression raises an AnalysisException()}}
{{---> 17   df}}
{{     18   .cube(f.col('col1'))}}
{{{}     19   .agg(f.grouping('col1') & f.lit(True)){}}}{{{}/databricks/spark/python/pyspark/sql/group.py in agg(self, *exprs){}}}
{{    116             # Columns}}
{{    117             assert all(isinstance(c, Column) for c in exprs), "all exprs should be Column"}}
{{--> 118             jdf = self._jgd.agg(exprs[0]._jc,}}
{{    119                                 _to_seq(self.sql_ctx._sc, [c._jc for c in exprs[1:]]))}}
{{{}    120         return DataFrame(jdf, self.sql_ctx){}}}{{{}/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in {_}{{_}}call{{_}}{_}(self, *args){}}}
{{   1302 }}
{{   1303         answer = self.gateway_client.send_command(command)}}
{{-> 1304         return_value = get_return_value(}}
{{   1305             answer, self.gateway_client, self.target_id, self.name)}}
{{   1306 }}{{/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)}}
{{    121                 # Hide where the exception came from that shows a non-Pythonic}}
{{    122                 # JVM exception message.}}
{{--> 123                 raise converted from None}}
{{    124             else:}}
{{{}    125                 raise{}}}{{{}AnalysisException: grouping() can only be used with GroupingSets/Cube/Rollup;{}}}
{{'Aggregate [cube(col1#548)|#548)], [col1#548, (grouping(col1#548) AND true) AS (grouping(col1) AND true)#551|#548, (grouping(col1#548) AND true) AS (grouping(col1) AND true)#551]}}
{{+- LogicalRDD [col1#548|#548], false}}
h1. Workaround

_Note:_ The reason I opened this ticket is that, when the user makes a particular type error, the resulting error message is misleading. The code snippet below shows how to fix that type error. It does not address the false-error-message bug, which is the focus of this ticket.

Cast the result of {{.grouping()}} to boolean type. That is, know _ab ovo_ that {{.grouping()}} produces an integer 0 or 1 rather than a boolean True or False.

{{(  # This expression does not raise an AnalysisException()}}
{{  df}}
{{  .cube(f.col('col1'))}}
{{  .agg(f.grouping('col1').cast(t.BooleanType()) & f.lit(True))}}
{{  .collect()}}
{{)}}
h1. Additional notes

The same error occurs if {{.cube()}} is replaced with {{.rollup()}} in "Code to reproduce".

The same error occurs if {{.grouping()}} is replaced with {{.grouping_id()}} in "Code to reproduce".
h1. Related tickets

https://issues.apache.org/jira/browse/SPARK-22748
h1. Relevant documentation
 * [Spark SQL GROUPBY, ROLLUP, and CUBE semantics|https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-groupby.html]
 * [DataFrame.cube()|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.cube.html]
 * [DataFrame.rollup()|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.rollup.html]
 * [DataFrame.agg()|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.agg.html]
 * [functions.grouping()|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.grouping.html]
 * [functions.grouping_id()|https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.grouping_id.html]