av9ash/GitBugs huggingface dataset:
Bug: Propagate correct JDBC properties in JDBC connector provider and add "connectionProvider" option
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 14:43
Affects Version/s: 3.1.0

There are a couple of issues with JDBC connection providers. The first is a bug caused by [https://github.com/apache/spark/commit/c3ce9701b458511255072c72b9b245036fa98653] where we would pass all properties, including JDBC data source keys, to the JDBC driver which results in errors like {{java.sql.SQLException: Unrecognized connection property 'url'}}.

Connection properties are supposed to only include vendor properties, url config is a JDBC option and should be excluded.

The fix would be replacing {{jdbcOptions.asProperties.asScala.foreach}} with {{jdbcOptions.asConnectionProperties.asScala.foreach}} which is java.sql.Driver friendly.

 

I also investigated the problem with multiple providers and I think there are a couple of oversights in {{ConnectionProvider}} implementation. I think it is missing two things:
 * Any {{JdbcConnectionProvider}} should take precedence over {{BasicConnectionProvider}}. {{BasicConnectionProvider}} should only be selected if there was no match found when inferring providers that can handle JDBC url.

 * There is currently no way to select a specific provider that you want, similar to how you can select a JDBC driver. The use case is, for example, having connection providers for two databases that handle the same URL but have slightly different semantics and you want to select one in one case and the other one in others.

 ** I think the first point could be discarded when the second one is addressed.

You can technically use {{spark.sql.sources.disabledJdbcConnProviderList}} to exclude ones that don’t need to be included, but I am not quite sure why it was done that way - it is much simpler to allow users to enforce the provider they want.

This ticket fixes it by adding a {{connectionProvider}} option to the JDBC data source that allows users to select a particular provider when the ambiguity arises.
Bug: 'spark.archives' does not extract the archive file into the driver under client mode
Status: Open | Priority: Major | Resolution: 
Created: 12/Jul/21 09:30
Affects Version/s: 3.1.2

When running spark in the k8s cluster, there are 2 deploy modes: cluster and client. After my test, in the cluster mode, *spark.archives* can extract the archive file to the working directory of the executors and driver. But in client mode, *spark.archives* can only extract the archive file to the working directory of the executors.

 

However, I need *spark.archives* to send the virtual environment tar file packaged by conda to both the driver and executors under client mode (So that the executor and the driver have the same python environment).

 

Why *spark.archives* does not extract the archive file into the working directory of the driver under client mode?
Bug: Disable CRAN in branches lower than the latest version uploaded
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/21 10:31
Affects Version/s: 3.0.3

CRAN check in branch-3.0 fails as below:

{code}
Insufficient package version (submitted: 3.0.4, existing: 3.1.2)
{code}

This is because CRAN doesn't allow lower version then the latest version. We can't upload so should better just skip the CRAN check.
Bug: unexpected Index loaded: pd.Index([10, 20, None], name="x")
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 12:06
Affects Version/s: 3.2.0

{code:python}
pidx = pd.Index([10, 20, 15, 30, 45, None], name="x")
psidx = ps.Index(pidx)
self.assert_eq(psidx.astype(str), pidx.astype(str))
{code}

[left pandas on spark]:  Index(['10.0', '20.0', '15.0', '30.0', '45.0', 'nan'], dtype='object', name='x')
[right pandas]: Index(['10', '20', '15', '30', '45', 'None'], dtype='object', name='x')

The index is loaded as float64, so the follow step like astype would be diff with pandas

[1] https://github.com/apache/spark/blob/bcc595c112a23d8e3024ace50f0dbc7eab7144b2/python/pyspark/pandas/tests/indexes/test_base.py#L2249
Bug: Refactor a few query compilation errors to use error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 23:13
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryCompilationErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala] to use error classes.

There are currently ~450 exceptions in this file, so this PR only refactors a few as an example.
Bug: Support creating a ps.Series/Index with `Decimal('NaN')` with Arrow disabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 21:19
Affects Version/s: 3.2.0

 
{code:java}
>>> import decimal as d
>>> import pyspark.pandas as ps
>>> import numpy as np
>>> ps.utils.default_session().conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> ps.Series([d.Decimal(1.0), d.Decimal(2.0), d.Decimal(np.nan)])
0       1
1       2
2    None
dtype: object
>>> ps.utils.default_session().conf.set('spark.sql.execution.arrow.pyspark.enabled', False)
>>> ps.Series([d.Decimal(1.0), d.Decimal(2.0), d.Decimal(np.nan)])
21/07/02 15:01:07 ERROR Executor: Exception in task 6.0 in stage 13.0 (TID 51)
net.razorvine.pickle.PickleException: problem construction object: java.lang.reflect.InvocationTargetException
...

{code}
As the code is shown above, we cannot create a Series with `Decimal('NaN')` when Arrow disabled. We ought to fix that.
Bug: hasnans for Series of Decimal(`NaN`)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 21:16
Affects Version/s: 3.2.0

{code:java}
>>> import pandas as pd
>>> pser = pd.Series([Decimal('0.1'), Decimal('NaN')])
>>> pser
0    0.1
1    NaN
dtype: object
>>> psser = ps.from_pandas(pser)
>>> psser
0     0.1
1    None
dtype: object
>>> psser.hasnans
False

{code}
Bug: decimal('Nan') is unsupported in net.razorvine.pickle 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 04:02
Affects Version/s: 3.2.0

Decimal('NaN') is not supported by net.razorvine.pickle now.

In Python
{code:java}
>>> pickled = cloudpickle.dumps(decimal.Decimal('NaN'))
b'\x80\x05\x95!\x00\x00\x00\x00\x00\x00\x00\x8c\x07decimal\x94\x8c\x07Decimal\x94\x93\x94\x8c\x03NaN\x94\x85\x94R\x94.'
>>> pickle.loads(pickled)
Decimal('NaN')
{code}
In Scala
{code:java}
scala> import net.razorvine.pickle.\{Pickler, Unpickler, PickleUtils}
scala> val unpickle = new Unpickler
scala> unpickle.loads(PickleUtils.str2bytes("\u0080\u0005\u0095!\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u008c\u0007decimal\u0094\u008c\u0007Decimal\u0094\u0093\u0094\u008c\u0003NaN\u0094\u0085\u0094R\u0094."))
net.razorvine.pickle.PickleException: problem construction object: java.lang.reflect.InvocationTargetException
 at net.razorvine.pickle.objects.AnyClassConstructor.construct(AnyClassConstructor.java:29)
 at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:773)
 at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:213)
 at net.razorvine.pickle.Unpickler.load(Unpickler.java:123)
 at net.razorvine.pickle.Unpickler.loads(Unpickler.java:136)
 ... 48 elided
{code}
I submit an issue in pickle upstream [https://github.com/irmen/pickle/issues/7] .

we should bump pickle latest version after it fixed.
Bug: CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 00:41
Affects Version/s: 3.2.0

Currently the config {{spark.sql.hive.metastorePartitionPruning}} is only used in {{PruneHiveTablePartitions}} but not {{PruneFileSourcePartitions}}. The latter calls {{CatalogFileIndex.filterPartitions}} which calls {{listPartitionsByFilter}} regardless of whether the above config is set or not.
Bug: UnwrapCastInBinaryComparison fail when in.list contain CheckOverflow expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 03:17
Affects Version/s: 3.2.0

UnwrapCastInBinaryComparison fail with the following case
 
{code:java}
CREATE TABLE tbl (d decimal(33, 27)) USING PARQUET 
SELECT d FROM tbl WHERE d NOT IN (d + 1){code}
 
Here {{in.list}} can be a {{CheckOverflow}} expression:
{code:java}
 cast(d#240 as decimal(34,27)) IN (CheckOverflow((promote_precision(cast(d#240 as decimal(34,27))) + 1.000000000000000000000000000), DecimalType(34,27), true)){code}
{{}}
Bug: Implement unary operator `invert` of integral ps.Series/Index
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/21 23:14
Affects Version/s: 3.2.0

 
{code:java}
>>> ~ps.Series([1, 2, 3])
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: cannot resolve '(NOT `0`)' due to data type mismatch: argument 1 requires boolean type, however, '`0`' is of bigint type.;
'Project [unresolvedalias(NOT 0#1L, Some(org.apache.spark.sql.Column$$Lambda$1365/2097273578@53165e1))]
+- Project [__index_level_0__#0L, 0#1L, monotonically_increasing_id() AS __natural_order__#4L]
 +- LogicalRDD [__index_level_0__#0L, 0#1L], false
{code}
 

 Currently, unary operator `invert` of integral ps.Series/Index is not supported. We ought to implement that following pandas' behaviors.
Bug: spark kafka offset missed some partition offset describle
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 12/Jul/21 02:57
Affects Version/s: 2.4.3

Caused by: org.apache.spark.sql.streaming.StreamingQueryException: Set(rcf_adidas_wow_room_event-9, rcf_adidas_wow_room_event-8, rcf_adidas_wow_room_event-7, rcf_adidas_wow_room_event-6, rcf_adidas_wow_room_event-5) are gone. Some data may have been missed.
 Some data may have been lost because they are not available in Kafka any more; either the
 data was aged out by Kafka or the topic may have been deleted before all the data in the
 topic was processed. If you don't want your streaming query to fail on such cases, set the
 source option "failOnDataLoss" to "false".
 === Streaming Query ===
 Identifier: [id = b0e0f54e-e638-430e-9912-751234a6ce13, runId = 570e950b-edcb-40c4-99ea-fd4db958b225]
 Current Committed Offsets: {KafkaV2[Subscribe[rcf_adidas_wow_room_event]]: {"rcf_adidas_wow_room_event":

{"8":3572,"2":3571,"5":3571,"4":3571,"7":3571,"1":3571,"9":3571,"3":3571,"6":3571,"0":3571}

}}
 Current Available Offsets: {KafkaV2[Subscribe[rcf_adidas_wow_room_event]]: {"rcf_adidas_wow_room_event":

{"2":3571,"4":3571,"1":3571,"3":3571,"0":3571}

}}

 

—

My topic has 10 partitions , but the last offset dir records only 5 partitions.

After i delete the last error offset dir , the query runs success.

 

But i do not understand what cause this error.
Bug: Spark driver requires large memory space for serialized results even there are no data collected to the driver
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 09/Jul/21 10:38
Affects Version/s: 2.4.3

Executing with large partition is causing the data transferred to driver exceed spark.driver.maxResultSize.

Even when no data from the logic is being collected at by the driver. Looks like spark is sending metadata back which is causing it to exceed.
{code:java}
spark.driver.maxResultSize=8g{code}
 
{code:java}
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 104904 tasks (8.0 GB) is bigger than spark.driver.maxResultSize (8.0 GB)Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 104904 tasks (8.0 GB) is bigger than spark.driver.maxResultSize (8.0 GB) at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2041) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2029) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2028) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2028) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:966) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:966) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:966) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2262) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2211) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2200) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:777) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114) at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78) ... 54 more{code}
Bug: Update the SQL migration guide about encoding auto-detection of CSV files 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 09:36
Affects Version/s: 3.0.4

Need to update the SQL migration guide to inform users about behavior change.

*What*: Spark doesn't detect encoding (charset) in CSV files with BOM correctly. Such files can be read only in the multiLine mode when the CSV option encoding matches to the actual encoding of CSV files. For example, Spark cannot read UTF-16BE CSV files when encoding is set to UTF-8 which is the default mode. This is the case of the current ES ticket.

*Why*: In previous Spark versions, encoding wasn't propagated to the underlying library that means the lib tried to detect file encoding automatically. It could success for some encodings that require BOM presents at the beginning of files. Starting from the versions 3.0, users can specify file encoding via the CSV option encoding which has UTF-8 as the default value. Spark propagates such default to the underlying library (uniVocity), and as a consequence this turned off encoding autodetection.

*When*: Since Spark 3.0. In particular, the commit https://github.com/apache/spark/commit/2df34db586bec379e40b5cf30021f5b7a2d79271 causes the issue.

*Workaround*: Enabling the encoding auto-detection mechanism in uniVocity by passing null as the value of CSV option encoding. A more recommended approach is to set the encoding option to UTF-16 explicitly.
Bug: Adjust `test_astype`, `test_neg` for old pandas versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/21 22:08
Affects Version/s: 3.2.0

* test_astype

For pandas < 1.1.0, declaring or converting to StringDtype was in general only possible if the data was already only str or nan-like (GH31204).

In pandas 1.1.0, the problem is adjusted by [https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.1.0.html#all-dtypes-can-now-be-converted-to-stringdtype].

That should be considered in `test_astype`, otherwise, current tests will fail with pandas < 1.1.0.
 * test_neg

{code:java}
dtypes = [
  "Int8",
  "Int16",
  "Int32",
  "Int64",
]
psers = []
for dtype in dtypes:
  psers.append(pd.Series([1, 2, 3, None], dtype=dtype))
  
for pser in psers:
  print((-pser).dtype){code}

 ~ 1.0.5, object dtype
 1.1.0~1.1.2, TypeError: bad operand type for unary -: 'IntegerArray'
 1.1.3, correct respective dtype
Bug:  Assume result's index to be disordered in tests with operations on different Series 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/21 22:39
Affects Version/s: 3.2.0

We have many tests with operations on different Series in spark/python/pyspark/pandas/tests/data_type_ops/ that assume the result's index to be sorted and then compare to the pandas' behavior. The assumption is wrong, so we should sort index of such result before comparing with pandas.
Bug: Try to capture faulthanlder when a Python worker crashes.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 21:13
Affects Version/s: 3.2.0

Currently, we just see an error message saying "exited unexpectedly (crashed)" when the UDFs causes the Python worker to crash by like segmentation fault.
 We should take advantage of {{[faulthandler|https://docs.python.org/3/library/faulthandler.html]}} and try to capture the error message from the {{faulthandler}}.
Bug: Migrate to GitHub Actions Codecov from Jenkins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 10:11
Affects Version/s: 3.2.0

We currently uses manual Codecov site to work around our Jenkins CI security issue. Now we use GitHub Actions so we can leverage Codecov to report the coverage for PySpark.

See also https://github.com/codecov/codecov-action
Bug: Add mlflow/sklearn to GHA docker image
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 10:52
Affects Version/s: 3.2.0

In Github Actions CI, we install the `mlflow>=1.0` and `sklearn` in the step "List Python packages (Python 3.9)" of "pyspark" job.

 

We can reduce the cost of CI by creating the image that has pre-installed both package.
Bug: AKS SPark UI does not have executor tab showing up
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/21 18:46
Affects Version/s: 2.2.0

Spark UI Executor tab showing blank and i see the below error in the network tab :

https://keplerfnet-aks-prod.az.3pc.att.com/proxy:10.128.0.76:4043/executors/

Failed to load resource: the server responded with a status of 404 ()

DevTools failed to load source map: Could not load content for [https://keplerfnet-aks-prod.az.3pc.att.com/proxy:10.128.0.76:4043/static/vis.map|https://ind01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkeplerfnet-aks-prod.az.3pc.att.com%2Fproxy%3A10.128.0.76%3A4043%2Fstatic%2Fvis.map&data=04%7C01%7CKoushik.Gopal%40TechMahindra.com%7C71ec48c8fa8d4ecc123908d95388dd8e%7Cedf442f5b9944c86a131b42b03a16c95%7C0%7C0%7C637632669559893674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UyrYVdDO4vfzwq4%2Fl4GHN6Gm8QC%2FMrvrGMl50FUCGrI%3D&reserved=0]: HTTP error: status code 502, net::ERR_HTTP_RESPONSE_CODE_FAILURE
Bug: Have Observation return Map instead of Row
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 19:37
Affects Version/s: 3.3.0

As [~gurwls223] pointed out, the {{Observation}} API (Scala, Java, PySpark) could return a {{Map}} / {{Dict}}. It currently returns {{Row}} simply because the metrics are (internal to {{Observation}}) retrieved from the listener as rows. Since that is hidden from the user by the {{Observation}} API, there is no need to return {{Row}}.

If there is some value in the original {{Row}}, both could be provided via {{getAsRow}} and {{getAsMap}}.

The {{Observation}} API has been added to Spark in unreleased 3.3.0, so it should not be a blocker to remove the {{Row}} return type in 3.3.0 again.
Bug: Remove old workarounds related to null ordering.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/21 22:15
Affects Version/s: 3.2.0

In pandas-on-Spark, there are still some remaining places to call {{Column._jc.(asc|desc)_nulls_(first|last)}} as a workaround from Koalas to support Spark 2.3.
Bug: Adjust `astype` method for ExtensionDtype in pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 16:44
Affects Version/s: 3.2.0

Currently, `astype` method for ExtensionDtype in pandas API on Spark is not consistent with pandas. For example, [https://github.com/apache/spark/pull/33095#discussion_r661704734.]

[https://github.com/apache/spark/pull/33095#discussion_r662623005.]

 

We ought to fill in the gap.
Bug: show api of Dataset should get as input the output method
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/21 18:55
Affects Version/s: 3.1.2

For now show is:
{code:scala}
def show(numRows: Int, truncate: Boolean): Unit = if (truncate) {
    println(showString(numRows, truncate = 20))
  } else {
    println(showString(numRows, truncate = 0))
  }
{code}
it can be turn into:
{code:scala}
def show(numRows: Int, truncate: Boolean, out: String => Unit = println): Unit = if (truncate) {
    out(showString(numRows, truncate = 20))
  } else {
    out(showString(numRows, truncate = 0))
  }
{code}
so user will be able to send that to file/log...
Bug: Move distributed-sequence implementation to Scala side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 04:48
Affects Version/s: 3.2.0

https://github.com/apache/spark/blob/c22f7a4834e6fb7b69c4cc4af87c61c2fbbe0786/python/pyspark/pandas/internal.py#L925-L945

This can be implemented in JVM side to make it more performance without extra serializations, and working around the nullability.
Bug: ThreadAudit test helper should log whether a thread is a Daemon thread
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 21:37
Affects Version/s: 3.1.2

It would be helpful if the POSSIBLE THREAD LEAK IN SUITE error mentioned whether the threads were daemon threads or not, since leaked non-daemon threads are more likely to be unintentional than leaked daemon threads.

https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/ThreadAudit.scala#L113
Bug: spark 3.1.1  version the  method  schema  of class  DataframeReader will lose the original type information of char/varchar
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 01/Jul/21 06:56
Affects Version/s: 3.1.1

in  spark 3.1.1 version,  the method schema of class DataframeReader will lose the original type information of char/varchar

 

!image-2021-07-01-14-54-49-922.png!

 

 

!image-2021-07-01-14-55-03-117.png!

 

 

It is expected that it can be placed in the metadata of the field, and the original type information can be obtained in subsequent processing
Bug: dropDuplicates does not type check argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 14:15
Affects Version/s: 3.1.2

When giving a single string to {{dropDuplicates}} method in pyspark a cryptic error is returned. Rather than returning a cryptic error, handle gracefully or return an exception. 

 

Proposal: Model after {{dropna}} behavior. If single string then use it to create a list, otherwise if argument is not list/tuple return an exception
Bug: Use check_exact=False in StatsTest.test_cov_corr_meta to loosen the check precision
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/21 22:20
Affects Version/s: 3.2.0

In some environment, the precision could be different in {{DataFrame.corr}} function.

We should use {{check_exact=False}} to loosen the precision.
Bug: Re-enable SPARK-32629 ShuffledHashJoin(full outer) metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/21 07:45
Affects Version/s: 3.2.0

After SPARK-36270, we ignore the test "{color:#6a8759}SPARK-32629: ShuffledHashJoin(full outer) metrics". We need to fix the test and re-enable it.
{color}
Bug: Fix hasnan() window function in IndexOpsMixin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/21 00:11
Affects Version/s: 3.2.0

 
{code:java}
File "/__w/spark/spark/python/pyspark/pandas/groupby.py", line 1497, in pyspark.pandas.groupby.GroupBy.rank
Failed example:
    df.groupby("a").rank().sort_index()
Exception raised:
...
pyspark.sql.utils.AnalysisException: It is not allowed to use a window function inside an aggregate function. Please use the inner window function in a sub-query.
{code}
As shown above, hasnans() used in "rank" causes "It is not allowed to use a window function inside an aggregate function" exception.

We shall adjust that.