Bug: Missing aggregate filter checks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/22 00:31
Affects Version/s: 3.4.0

h3. Window function in filter
{noformat}
select sum(a) filter (where nth_value(a, 2) over (order by b) > 1)
from (select 1 a, '2' b);
{noformat}
This query should produce an analysis error, but instead produces a stack overflow:
{noformat}
java.lang.StackOverflowError: null
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$collect$1(TreeNode.scala:305) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$collect$1$adapted(TreeNode.scala:305) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:264) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:265) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:265) ~[spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
	at scala.collection.Iterator.foreach(Iterator.scala:943) ~[scala-library.jar:?]
...
{noformat}
h3. Non-boolean filter expression
{noformat}
select sum(a) filter (where a) from (select 1 a, '2' b);
{noformat}
This query should produce an analysis error, but instead causes a projection compilation error or whole-stage codegen error (depending on the datatype of the expression):
{noformat}
22/03/26 17:19:03 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 50, Column 6: Not a boolean expression
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 50, Column 6: Not a boolean expression
	at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12021) ~[janino-3.0.16.jar:?]
	at org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4049) ~[janino-3.0.16.jar:?]
	at org.codehaus.janino.UnitCompiler.access$6300(UnitCompiler.java:226) ~[janino-3.0.16.jar:?]
	at org.codehaus.janino.UnitCompiler$14.visitIntegerLiteral(UnitCompiler.java:4016) ~[janino-3.0.16.jar:?]
...
22/03/26 17:19:05 WARN MutableProjection: Expr codegen error and falling back to interpreter mode
java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 15: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 40, Column 15: Not a boolean expression
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306) ~[guava-14.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293) ~[guava-14.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-14.0.1.jar:?]
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135) ~[guava-14.0.1.jar:?]
	at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410) ~[guava-14.0.1.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2380) ~[guava-14.0.1.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) ~[guava-14.0.1.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257) ~[guava-14.0.1.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4000) ~[guava-14.0.1.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004) ~[guava-14.0.1.jar:?]
...
NULL
Time taken: 5.397 seconds, Fetched 1 row(s)
{noformat}
Interestingly, it also returns a result (NULL).
h3. Aggregate expression in filter expression
{noformat}
select max(b) filter (where max(a) > 1) from (select 1 a, '2' b);
{noformat}
This query should produce an analysis error, but instead causes a projection compilation error or whole-stage codegen error (depending on the datatype of the expression being aggregated):
{noformat}
22/03/26 17:26:38 ERROR TaskSetManager: Task 0 in stage 3.0 failed 1 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 2) (10.0.0.106 executor driver): org.apache.spark.SparkUnsupportedOperationException: Cannot evaluate expression: max(1)
	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotEvaluateExpressionError(QueryExecutionErrors.scala:79)
	at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval(Expression.scala:344)
	at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval$(Expression.scala:343)
	at org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.eval(interfaces.scala:99)
	at org.apache.spark.sql.catalyst.expressions.BinaryExpression.eval(Expression.scala:593)
	at org.apache.spark.sql.catalyst.expressions.If.eval(conditionalExpressions.scala:68)
...
{noformat}