av9ash/GitBugs huggingface dataset:
Bug: HadoopVersionInfoSuite faild when maven test in Scala 2.13
Status: Closed | Priority: Minor | Resolution: Not A Problem
Created: 13/Jan/21 12:08
Affects Version/s: 3.2.0

 
{code:java}
mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -pl sql/hive -Pscala-2.13  -DwildcardSuites=org.apache.spark.sql.hive.client.HadoopVersionInfoSuite -Dtest=none 
{code}
Independent test HadoopVersionInfoSuite all case passed, but execute

 
{code:java}
mvn clean test -Phadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -pl sql/hive -Pscala-2.13 
{code}
 

HadoopVersionInfoSuite failed as follow:
{code:java}
HadoopVersionInfoSuite: 22:32:30.310 WARN org.apache.spark.sql.hive.client.IsolatedClientLoader: Failed to resolve Hadoop artifacts for the version 2.7.4. We will change the hadoop version from 2.7.4 to 2.7.4 and try again. It is recommended to set jars used by Hive metastore client through spark.sql.hive.metastore.jars in the production environment. - SPARK-32256: Hadoop VersionInfo should be preloaded *** FAILED *** java.lang.RuntimeException: [unresolved dependency: org.apache.hive#hive-metastore;2.0.1: not found, unresolved dependency: org.apache.hive#hive-exec;2.0.1: not found, unresolved dependency: org.apache.hive#hive-common;2.0.1: not found, unresolved dependency: org.apache.hive#hive-serde;2.0.1: not found, unresolved dependency: com.google.guava#guava;14.0.1: not found, unresolved dependency: org.apache.hadoop#hadoop-client;2.7.4: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1423) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.$anonfun$downloadVersion$2(IsolatedClientLoader.scala:122) at org.apache.spark.sql.catalyst.util.package$.quietly(package.scala:42) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.downloadVersion(IsolatedClientLoader.scala:122) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.liftedTree1$1(IsolatedClientLoader.scala:75) at org.apache.spark.sql.hive.client.IsolatedClientLoader$.forVersion(IsolatedClientLoader.scala:63) at org.apache.spark.sql.hive.client.HadoopVersionInfoSuite.$anonfun$new$1(HadoopVersionInfoSuite.scala:46) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
{code}
need some investigate.
Bug: Linear Discriminant Analysis for dimensionality reduction
Status: Open | Priority: Minor | Resolution: 
Created: 07/Jan/21 16:51
Affects Version/s: 3.0.1

The idea is to implement Linear discriminant analysis for dimensionality reduction. The algorithm is similar to PCA but it uses supervised classification to maximize class separation.  The API would be similar to the PCA one.

 

Other frameworks implement LDA, like sklearn ([https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html)|http://example.com/]

 

LDA is a well-known algorithm with related literature, for example:

Izenman, Alan Julian. "Linear discriminant analysis." _Modern multivariate statistical techniques_. Springer, New York, NY, 2013. 237-280.

 

I would like to work on this issue, please let me know if it is interesting to add this feature to spark.

 

Thank you in advance.
Bug: Separate state store numKeys metric test
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jan/21 18:15
Affects Version/s: 3.1.1

Right now in StateStoreSuite, the tests of get/put/remove/commit are mixed with numKeys metric test. I found it is flaky when I was testing with other StateStore implementation. Specifically, we also are able to check these metrics after state store is updated (committed).  So I think we can refactor the test a little bit to make it easier to incorporate other StateStore externally.
Bug: Queries that only touch partition columns shouldn't scan through all files
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 21/Jan/21 19:31
Affects Version/s: 3.2.0

When querying only the partition columns of a partitioned table, it seems that Spark nonetheless scans through all files in the table, even though it doesn't need to.

Here's an example:
{code:python}
>>> data = spark.read.option('mergeSchema', 'false').parquet('s3a://some/dataset')
[Stage 0:==================>                                  (407 + 12) / 1158]
{code}
Note the 1158 tasks. This matches the number of partitions in the table, which is partitioned on a single field named {{file_date}}:
{code:sh}
$ aws s3 ls s3://some/dataset | head -n 3
                           PRE file_date=2017-05-01/
                           PRE file_date=2017-05-02/
                           PRE file_date=2017-05-03/

$ aws s3 ls s3://some/dataset | wc -l
    1158
{code}
The table itself has over 138K files, though:
{code:sh}
$ aws s3 ls --recursive --human --summarize s3://some/dataset
...
Total Objects: 138708
   Total Size: 3.7 TiB
{code}
Now let's try to query just the {{file_date}} field and see what Spark does.
{code:python}
>>> data.select('file_date').orderBy('file_date', ascending=False).limit(1).explain()
== Physical Plan ==
TakeOrderedAndProject(limit=1, orderBy=[file_date#11 DESC NULLS LAST], output=[file_date#11])
+- *(1) ColumnarToRow
   +- FileScan parquet [file_date#11] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[s3a://some/dataset], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>

>>> data.select('file_date').orderBy('file_date', ascending=False).limit(1).show()
[Stage 2:>                                                   (179 + 12) / 41011]
{code}
Notice that Spark has spun up 41,011 tasks. Maybe more will be needed as the job progresses? I'm not sure.

What I do know is that this operation takes a long time (~20 min) running from my laptop, whereas to list the top-level {{file_date}} partitions via the AWS CLI take a second or two.

Spark appears to be going through all the files in the table, when it just needs to list the partitions captured in the S3 "directory" structure. The query is only touching {{file_date}}, after all.

The current workaround for this performance problem / optimizer wastefulness, is to [query the catalog directly|https://stackoverflow.com/a/65724151/877069]. It works, but is a lot of extra work compared to the elegant query against {{file_date}} that users actually intend.

Spark should somehow know when it is only querying partition fields and skip iterating through all the individual files in a table.

Tested on Spark 3.0.1.
Bug: Fix flaky test in DecommissionWorkerSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/21 04:05
Affects Version/s: 3.1.0

The test `decommission workers ensure that shuffle output is regenerated even with shuffle service` assumes it has two executor and both of two tasks can execute concurrently.

The two tasks will execute serially if there only one executor. The result is test is unexpceted. E.g. 

```

[info] 5 did not equal 4 Expected 4 tasks but got List(0:0:0:0-SUCCESS, 0:0:1:0-FAILED, 0:0:1:1-SUCCESS, 0:1:0:0-SUCCESS, 1:0:0:0-SUCCESS) (DecommissionWorkerSuite.scala:190)

```
Bug: Reading a partitioned dataset with a partition value of NOW causes the value to be parsed as a timestamp.
Status: In Progress | Priority: Minor | Resolution: 
Created: 27/Jan/21 09:03
Affects Version/s: 3.0.1

*Problem*

Reading a partitioned dataset where one of the column values matches a special timestamp (NOW, TODAY etc) causes the value to be interpreted as a timestamp rather than a string. 

*Example Code (Scala)*
{code:java}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object TestBug {

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[*]").getOrCreate()

    val df = spark.range(1, 2).withColumn("partition", lit("NOW"))
    df.write.mode("overwrite").partitionBy("partition").parquet("bug")
    
    spark.read.parquet("bug").show(truncate = false)
  }

}
{code}
 The above program prints out:
{noformat}
+---+--------------------------+
|id |partition |
+---+--------------------------+
|1 |2021-01-27 08:53:23.650039|
+---+--------------------------+
{noformat}
 

*Analysis*

This happens because in PartitioningUtils.inferPartitionColumnValue we try and cast the partition value to a timestamp in order to determine if timestamp is a valid interpretation.  As NOW etc are literals which are valid to cast to timestamps, the code ends up as interpreting the value as a timestamp.

I think what we want to do here is change PartitioningUtils.inferPartitionColumnValue so that when it  attempts to interpret as timestamp we ignore the special values. This looks difficult to do if we continue to use cast, but one other option is to add an option to
DateTimeUtils.stringToDate to tell it to ignore special values and instead use that to do the conversion.
Bug: Add a tip info for the `resources` column in the executors page
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jan/21 08:08
Affects Version/s: 3.0.1

After upgrading from spark 2.3 to spark 3.0.1, the *Resources* column in the executors page is always blank because it does not use GPU/FPGA, 
 and there is no tip info, so users are often confused when they do not know the exact meaning of this column.
Bug: Memory-size-based object hash aggregation fallback mechanism
Status: Open | Priority: Minor | Resolution: 
Created: 29/Jan/21 06:27
Affects Version/s: 3.2.0

As object hash aggregate fallback mechanism is special - it will fallback to sort-based aggregation based on number of keys seen so far [0]. This fallback logic sometimes is sub-optimal and leads to unnecessary sort, and performance degradation in run-time. Given the on-heap object hash aggregation [map value is a general {{InternalRow}}|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationMap.scala#L36], it is hard to get an accurate metrics for memory size of this map. Hive aggregation uses a combination of [current JVM heap memory usage|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java#L872], and [estimation of map size|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java#L919-L920] to decide whether to spill. Hive's approach might be better. I am also tagging queries in our production environment with object hash aggregate fallback, and see how to improve them. One thing we can do here is to monitor JVM heap memory usage, and estimate the on-heap object hash aggregation map size. Fallback based on the information of above, instead of number of keys.

[0]: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala#L172]
Bug: Add countDistinct option to Dataset#summary
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/21 02:43
Affects Version/s: 3.2.0

The Dataset#summary function supports options like count, mean, min, and max.  It's a great little function for lightweight exploratory data analysis.

A count distinct of each column is a common exploratory data analysis workflow.  This should be easy to add (piggybacking off the existing countDistinct code), entirely backwards compatible, and will help a lot of users.
Bug: In a special scenario, the error message of the stage in the UI page is blank
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/21 04:34
Affects Version/s: 3.0.1

version: spark 3.0.1
problem: as shown in the attchement,  When we visited Spark's historical task information UI page, we found that many stages of the current task failed, but some stages displayed correct error messages, while others were left blank
Bug: Prevent incomplete master URLs for Spark on Kubernetes early
Status: Open | Priority: Minor | Resolution: 
Created: 27/Jan/21 13:40
Affects Version/s: 3.0.1

It turns out that {{--master k8s://}} is accepted and although leads to termination displays stacktraces that don't really tell what the real cause is.

This may happen when the Kubernetes API server(s) are described by an environment variable that's not initialized in the current terminal.

{code}
$ ./bin/spark-shell --master k8s:// --verbose
...
Spark config:
(spark.jars,)
(spark.app.name,Spark shell)
(spark.submit.pyFiles,)
(spark.ui.showConsoleProgress,true)
(spark.submit.deployMode,client)
(spark.master,k8s://https://)
...
21/01/27 14:29:44 ERROR Main: Failed to initialize Spark session.
io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:208)
...
Caused by: java.net.UnknownHostException: api: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at okhttp3.Dns$1.lookup(Dns.java:40)
	at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
{code}
Bug: Fix DockerJDBCIntegrationSuites to reflect the change of SPARK-33888
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Jan/21 05:21
Affects Version/s: 3.2.0

After SPARK-33888, the JDBC type `sql.Types.TIME` is mapped to the catalyst type `sql.types.IntegerType` but DockerJDBCIntegrationSuites doesn't reflect the change.
Bug: Let HiveExternalCatalogVersionsSuite can run in orgs internal environment 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jan/21 11:12
Affects Version/s: 3.2.0

HiveExternalCatalogVersionsSuite can't run in orgs internal environment where access to outside internet is not allowed because HiveExternalCatalogVersionsSuite will download spark release package in suite.
Bug: Base value for parsing two-digit year should be made configurable
Status: Open | Priority: Minor | Resolution: 
Created: 21/Jan/21 19:57
Affects Version/s: 3.0.1

The base value is set as 2000 within spark for parsing a two-digit year date string. If we try to parse "10-JAN-97" then this will be interpreted as 2097 instead of 1997.

I'm unclear as to why this base value has been changed within spark as the standard python datetime module instead uses a more sensible value of 69 as the boundary cut-off for determining the correct century to apply.

Reference: [https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html]

Other libraries e.g. .NET Core will use 29 as the boundary cut-off. But a base value of 2000 is rather non-functional indeed. Most dates encountered in the real world will pertain to both centuries and therefore I propose this functionality is reverted to match the existing python datetime module and / or allow the base value to be set as an option to the various date functions. This would ensure there's consistent behaviour across both python and pyspark.

 

Python:

{{import datetime}}
 {{datetime.datetime.strptime('10-JAN-69', '%d-%b-%y').date()}}
 Out[118]: datetime.date(1969, 1, 10)
  
 {{Pyspark:}}{{import spark.sql.functions as F}}
 {{df = spark.createDataFrame([('10-JAN-69',)], ['dt'])}}
 {{df.withColumn("newdate", F.from_unixtime(F.unix_timestamp("dt", "dd-MMM-yy"), "dd-MM-yyyy")).collect()}}
  
 Out[117]: [Row(dt='10-JAN-69', newdate='10-01-2069')]
  
  
  
 As a work around I had to write my own solution to deal with this. The code below is specific to my data pipeline but you get the idea of the issue I had to deal with just to change the boundary cut-off to better handle two-digit years.
  
 {{from pyspark.sql.functions import to_date, col, trim}}{{def convert_dtypes(entity, schema, boundary="40"):}}
 \{{ cols = []}}
 {{ for x in schema[entity]:}}
 \{{ for c in std_df.columns:}}
 {{ if x['name'] == c:}}
 {{ if x['dtype'] == 'date':}}
 \{{ dd = F.substring(c, 1, 2)}}
 \{{ MMM = F.substring(c, 4, 3)}}
 \{{ yy = F.substring(c, 8, 2)}}
 \{{ n = (}}
 \{{ F.when(trim(col(c)) == "", None).otherwise(}}
 \{{ F.when(yy >= ("40"), }}
 {{ F.concat(dd, F.lit("-"), MMM, F.lit("-"), F.lit("19"), yy)).otherwise(}}
 {{ F.concat(dd, F.lit("-"), MMM, F.lit("-"), F.lit("20"), yy)))}}
 \{{ )}}
 \{{ cols.append(to_date(n, 'dd-MMM-yyyy').alias(c))}}
 \{{ else:}}
 {{ cols.append(col(c).cast(x['dtype']))}}
 {{ #cols[-1].nullable = x['nullable']}}
 \{{ return std_df.select(*cols)}}
Bug: Java UT failed when testing sql/hive module independently in Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Jan/21 07:33
Affects Version/s: 3.2.0

When execute mvn test of sql/hive module independently in Scala 2.13, there is Java UT failed as follow: 
 # 
{code:java}
mvn clean install -Dhadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl  sql/hive -am -DskipTests{code}

 # 
{code:java}
mvn test -Dhadoop-3.2 -Phive-2.3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Pscala-2.13 -pl  sql/hive{code}

 # 
{code:java}
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 20.353 s <<< FAILURE! - in org.apache.spark.sql.hive.JavaDataFrameSuite
[ERROR] org.apache.spark.sql.hive.JavaDataFrameSuite.testUDAF  Time elapsed: 18.548 s  <<< ERROR!
java.lang.NoClassDefFoundError: scala/collection/parallel/TaskSupport
	at org.apache.spark.sql.hive.JavaDataFrameSuite.checkAnswer(JavaDataFrameSuite.java:41)
	at org.apache.spark.sql.hive.JavaDataFrameSuite.testUDAF(JavaDataFrameSuite.java:92)
Caused by: java.lang.ClassNotFoundException: scala.collection.parallel.TaskSupport
	at org.apache.spark.sql.hive.JavaDataFrameSuite.checkAnswer(JavaDataFrameSuite.java:41)
	at org.apache.spark.sql.hive.JavaDataFrameSuite.testUDAF(JavaDataFrameSuite.java:92)
{code}
Bug: Move QueryCompilationErrors.scala and QueryExecutionErrors.scala to org/apache/spark/sql/errors
Status: In Progress | Priority: Minor | Resolution: 
Created: 16/Jan/21 20:59
Affects Version/s: 3.2.0

QueryCompilationErrors.scala and QueryExecutionErrors.scala use the org.apache.spark.sql.errors package, but these files are reside in org/apache/spark/sql directory.
Bug: Remove duplicated branches in case when
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Jan/21 05:10
Affects Version/s: 3.2.0

CaseWhen with duplicated branches could be dedup
{code}
SELECT CASE WHEN key = 1 THEN 1 WHEN key = 1 THEN 1 WHEN key = 1 THEN 1
ELSE 2 END FROM testData WHERE key = 1 group by key
{code}
Bug: Strip auto-generated cast when using Cast.sql
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jan/21 11:48
Affects Version/s: 3.2.0

During analysis we may introduce the Cast if exists type cast implicitly. That makes assgined name unclear.

Let's say we have a sql `select id == null` which id is int type, then the output field name will be `(id = CAST(null as int))`.
Bug: Spark SQL cannot escape both \ and other special characters 
Status: Open | Priority: Minor | Resolution: 
Created: 13/Jan/21 22:07
Affects Version/s: 2.0.2

Spark literal string parsing does not properly escape backslashes or other special characters. This is an extension of this issue: https://issues.apache.org/jira/browse/SPARK-17647#

 

The issue is that depending on how spark.sql.parser.escapedStringLiterals is set, you will either be able to correctly get escaped backslashes in a string literal, but not escaped other special characters, OR, you can have correctly escaped other special characters, but not correctly escaped backslashes.

So you have to choose which configuration you care about more.

I have tested Spark versions 2.1, 2.2, 2.3, 2.4, and 3.0 and they all experience the issue:
{code:java}
# These do not return the expected backslash
SET spark.sql.parser.escapedStringLiterals=false;
SELECT '\\';
> \
(should return \\)

SELECT 'hi\hi';
> hihi
(should return hi\hi) 


# These are correctly escaped
SELECT '\"';
> "

 SELECT '\'';
> '{code}
If I switch this: 
{code:java}
# These now work
SET spark.sql.parser.escapedStringLiterals=true;
SELECT '\\';
> \\

SELECT 'hi\hi';
> hi\hi


# These are now not correctly escaped
SELECT '\"';
> \"
(should return ")

SELECT '\'';
> \'
(should return ' ){code}
 So basically we have to choose:

SET spark.sql.parser.escapedStringLiterals=false; if we want backslashes correctly escaped but not other special characters

SET spark.sql.parser.escapedStringLiterals=true; if we want other special characters correctly escaped but not backslashes
Bug: PartitionPruning push down pruningHasBenefit function into insertPredicate function to decrease calculate time
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Jan/21 07:15
Affects Version/s: 3.0.0

in the class PartitionPruning, function prune, push pruningHasBenefit function into insertPredicate function, as `SQLConf.get.exchangeReuseEnabled` is always  true default and `SQLConf.get.dynamicPartitionPruningReuseBroadcastOnly `  is always true default.

by set hasBenefit to lazy ,we can do not need invoke hasBenefit ,so we can save time.

solved by #31122
Bug: resolveOperatorsUpWithNewOutput should wrap allowInvokingTransformsInAnalyzer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Jan/21 13:01
Affects Version/s: 3.1.0

PaddingAndLengthCheckForCharVarchar could fail query when resolveOperatorsUpWithNewOutput
with 


{code:java}
[info] - char/varchar resolution in sub query  *** FAILED *** (367 milliseconds)
[info]   java.lang.RuntimeException: This method should not be called in the analyzer
[info]   at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.assertNotAnalysisRule(AnalysisHelper.scala:150)
[info]   at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.assertNotAnalysisRule$(AnalysisHelper.scala:146)
[info]   at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.assertNotAnalysisRule(LogicalPlan.scala:29)
[info]   at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDown(AnalysisHelper.scala:161)
[info]   at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDown$(AnalysisHelper.scala:160)
[info]   at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
[info]   at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
[info]   at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$updateOuterReferencesInSubquery(QueryPlan.scala:267)
{code}
Bug: Window expression with alias inside WHERE and HAVING clauses fail with non-descriptive exceptions
Status: Closed | Priority: Minor | Resolution: Invalid
Created: 12/Jan/21 05:18
Affects Version/s: 3.0.2

SPARK-24575 prohibits window expressions inside WHERE and HAVING clauses. But if the window expression with alias inside WHERE and HAVING clauses, Spark does not handle this explicitly and will fail with non-descriptive exceptions.
{code}
SELECT a, RANK() OVER(ORDER BY b) AS s FROM testData2 WHERE b = 2 AND s = 1
{code}
{code}
cannot resolve '`s`' given input columns: [testdata2.a, testdata2.b]
{code}
{code}
SELECT a, MAX(b), RANK() OVER(ORDER BY a) AS s
FROM testData2
GROUP BY a
HAVING SUM(b) = 5 AND s = 1
{code}
{code}
cannot resolve '`b`' given input columns: [testdata2.a, max(b)]
{code}