av9ash/GitBugs huggingface dataset:
Bug: Avoid redundant ColumnarToRow transistion on InMemoryTableScan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/21 04:48
Affects Version/s: 3.3.0

We have a rule to insert columnar transition between row-based and columnar query plans. InMemoryTableScanExec can produce columnar output. So if its parent plan isn't columnar, the rule adds a ColumnarToRow between them.

But InMemoryTableScanExec is a special query plan because it can convert from cached batch to columnar batch or row. 

For such case, we ask InMemoryTableScanExec to convert cached batch to columnar batch, and then convert to row in the added ColumnarToRow, before the parent query.

So for such case, we can simply ask InMemoryTableScanExec to produce row output instead of a redundant conversion.

```
               +- Union                                                                                                                                                      
                  :- ColumnarToRow                                                                                                                                           
                  :  +- InMemoryTableScan [i#8, j#9]                                                                                                                         
                  :        +- InMemoryRelation [i#8, j#9], StorageLevel(disk, memory, deserialized, 1 replicas)     
```
Bug: Supports write data flow control
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/21 03:15
Affects Version/s: 3.2.0

The hive table data is written to a relational database, generally an online production database. If the writing speed has no traffic control, it can easily affect the stability of the online system. It is recommended to add traffic control parameters

[~fanjia]
Bug: Flay test: HealthTrackerIntegrationSuite.If preferred node is bad, without excludeOnFailure job will fail
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 04:40
Affects Version/s: 3.3.0

https://github.com/apache/spark/runs/4259932001

{code}
sbt.ForkMain$ForkError: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:187)
	at org.apache.spark.util.ThreadUtils$.awaitReady(ThreadUtils.scala:334)
	at org.apache.spark.scheduler.SchedulerIntegrationSuite.awaitJobTermination(SchedulerIntegrationSuite.scala:274)
	at org.apache.spark.scheduler.HealthTrackerIntegrationSuite.$anonfun$new$3(HealthTrackerIntegrationSuite.scala:50)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.scheduler.SchedulerIntegrationSuite.withBackend(SchedulerIntegrationSuite.scala:261)
	at org.apache.spark.scheduler.HealthTrackerIntegrationSuite.$anonfun$new$1(HealthTrackerIntegrationSuite.scala:48)
	at org.apache.spark.scheduler.SchedulerIntegrationSuite.$anonfun$testScheduler$1(SchedulerIntegrationSuite.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
{code}
Bug: hive-2.3.9 related API use invoke method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/21 07:31
Affects Version/s: 3.2.0

Here is a condition that when hive version >= 2.3.9, it will call
{code}
case (2, 3, v) if v >= 9 => Hive.getWithoutRegisterFns(conf)
{code}

This method means to use  getWithoutRegisterFns in 2.3.9 but if we use code of 2.3.8 or lower version, this will cause build failed, we should use invoke method here
Bug: ALTER TABLE .. REPLACE COLUMNS works incorrectly for v2 tables in tests
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Nov/21 10:51
Affects Version/s: 3.3.0

The test demonstrates the issue:

{code:scala}
  test("Replace columns in v2 table") {
    withNamespaceAndTable("ns", "tbl") { t =>
      sql(s"CREATE TABLE $t (i INT, data STRING) $defaultUsing")
      sql(s"INSERT INTO $t SELECT 0, 'abc'")
      sql(
        s"""
           |ALTER TABLE $t REPLACE COLUMNS (
           | new_i STRING,
           | new_data LONG)""".stripMargin)
      sql(s"INSERT INTO $t SELECT 'def', 1000")

      checkAnswer(
        sql(s"SELECT new_i, new_data FROM $t"),
        Seq(Row("def", 1000)))
    }
  }
{code}

The test fails:
{code:java}
== Results ==
!== Correct Answer - 1 ==   == Spark Answer - 2 ==
!struct<>                   struct<new_i:string,new_data:bigint>
![def,1000]                 [,103079215107]
!                           [def,1000]
{code}
Bug: Upgrade Avro to 1.11.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/21 04:26
Affects Version/s: 3.3.0

Recently, Avro 1.1.0 was released which includes bunch of bug fixes.
https://issues.apache.org/jira/issues/?jql=project%3DAVRO%20AND%20fixVersion%3D1.11.0
Bug: Avoid unnecessary sort in FileFormatWriter if it's not dynamic partition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Nov/21 08:22
Affects Version/s: 3.3.0

`FileFormatWriter.write` will sort the partition and bucket column before writing. I think this code path assumed the input `partitionColumns` are dynamic but actually it's not. It now is used by three code path:
 - `FileStreamSink`; it should be always dynamic partition
 - `SaveAsHiveFile`; it followed the assuming that `InsertIntoHiveTable` has removed the static partition and `InsertIntoHiveDirCommand` has no partition
 - `InsertIntoHadoopFsRelationCommand`; it passed `partitionColumns` into `FileFormatWriter.write` without removing static partition because we need it to generate the partition path in `DynamicPartitionDataWriter`

It shows that the unnecessary sort only affected the `InsertIntoHadoopFsRelationCommand` if we write data with static partition.
Bug: Result vector from pandas_udf was not the required length
Status: Closed | Priority: Major | Resolution: Invalid
Created: 15/Nov/21 03:41
Affects Version/s: 3.2.0

 
{code:java}
schema = StructType([
StructField("node", StringType())
])
rdd = sc.textFile("hdfs:///user/liubiao/KG/graph_dict.txt")
rdd = rdd.map(lambda obj: {'node': obj})
df_node = spark.createDataFrame(rdd, schema=schema)

df_fname =spark.read.parquet("hdfs:///user/liubiao/KG/fnames.parquet")
pd_fname = df_fname.select('fname').toPandas()

@pandas_udf(IntegerType(), PandasUDFType.SCALAR)
def udf_match(word: pd.Series) -> pd.Series:
  my_Series = pd_fname.squeeze() # dataframe to Series
  num = int(my_Series.str.contains(word.array[0]).sum())
  return pd.Series(num)

df = df_node.withColumn("match_fname_num", udf_match(df_node["node"]))
{code}
 

Hi, I have two dataframe, and I try above method, however, I get this
{code:java}
RuntimeError: Result vector from pandas_udf was not the required length: expected 100, got 1{code}
it will be really thankful, if there is any helps

 

PS: for the method itself, I think there is no problem, I create same sample data to verify it successfully, however, when I use the real data error came. I checked the data, can't figure out,

does anyone know what it cause?
Bug: pyspark.pandas histogram accepts the title option but does not add a title to the plot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/21 21:16
Affects Version/s: 3.2.0

In pyspark.pandas if you write a line like this
{quote}DF.plot.hist(bins=20, title="US Counties -- FullVaxPer100")
{quote}
it compiles and runs, but the plot has no title.
Bug: Side effects between PySpark Pandas UDF and Numpy indexing
Status: Open | Priority: Major | Resolution: 
Created: 23/Nov/21 12:12
Affects Version/s: 3.1.2

Let's create a simple Pandas Dataframe with a single column named 'id' that contains a sequential range.
{code:java}
df = pd.DataFrame(np.arange(0,1000), columns=['id']){code}
Consider this function that selects the first 4 indexes of the 'id' column of an array.
{code:java}
def udf_example(df):
  
  some_index = np.array([0, 1, 2, 3])
  values = df['id'].values[some_index]
  
  df = pd.DataFrame(values, columns=['id'])
  return df{code}
If I apply this function in Pyspark I get this result:
{code:java}
schema = t.StructType([t.StructField('id', t.LongType(), True)])
df_spark = spark.createDataFrame(df).groupBy().applyInPandas(udf_example, schema)
display(df_spark)
# id
# 125
# 126
# 127
# 128
{code}
If I apply it in Python I get the correct and expected result:
{code:java}
udf_example(df)
# id
# 0
# 1
# 2
# 3
{code}
Using NumPy indexing operations inside a Pandas UDF in Spark causes side effects and unexpected results.
Bug: Spark SQL reads unnecessary nested fields (another type of pruning case)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/21 12:22
Affects Version/s: 3.2.0

Based on this [SPARK-34638|https://issues.apache.org/jira/browse/SPARK-34638] Maybe I found another nested fields pruning case. In this case I found full read with `count` function

Example:
1) Loading data

{code:scala}
val jsonStr = """{
 "items": [
   {"itemId": 1, "itemData": "a"},
   {"itemId": 2, "itemData": "b"}
 ]
}"""
val df = spark.read.json(Seq(jsonStr).toDS)
df.write.format("parquet").mode("overwrite").saveAsTable("persisted")
{code}

2) read query with explain

{code:scala}
val read = spark.table("persisted")
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", true)

read.select(explode($"items").as('item)).select(count(lit(true))).explain(true)
// ReadSchema: struct<items:array<struct<itemData:string,itemId:bigint>>>
{code}
Bug: SIGNIFICANT bottleneck introduced by fix for SPARK-32001
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 19:27
Affects Version/s: 3.1.0

The fix for https://issues.apache.org/jira/browse/SPARK-32001 ( [https://github.com/apache/spark/pull/29024/files#diff-345beef18081272d77d91eeca2d9b5534ff6e642245352f40f4e9c9b8922b085R58] ) does not seem to have consider the reality that some apps may rely on being able to establish many JDBC connections simultaneously for performance reasons.

The fix forces concurrency to 1 when establishing database connections and that strikes me as a *significant* user impacting change and a *significant* bottleneck.

Can anyone propose a workaround for this? I have an app that makes connections to thousands of databases and I can't upgrade to any version >3.1.x because of this significant bottleneck.

 

Thanks in advance for your help!
Bug: Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/21 19:28
Affects Version/s: 3.3.0

- Undeprecate (Spark)DataFrame.to_koalas 

- Deprecate (Spark)DataFrame.to_pandas_like and introduce (Spark)DataFrame.pandas_api instead.
Bug: EventLoggingListener keep logging errors after hdfs restart all datanodes
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/21 03:10
Affects Version/s: 2.4.0

I saw the error in SparkThriftServer process's log when I restart all datanodes of HDFS , The logs as follows:

{code:java}
2021-11-16 13:52:11,044 ERROR [spark-listener-group-eventLog] scheduler.AsyncEventQueue:Listener EventLoggingListener threw an exception
java.io.IOException: All datanodes [DatanodeInfoWithStorage[10.121.23.101:1019,DS-90cb8066-8e5c-443f-804b-20c3ad01851b,DISK]] are bad. Aborting...
at org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1561)
at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1495)
at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
at org.apache.hadoop.hdfs.DataStreamer.processDatanodeErrorOrExternalError(DataStreamer.java:1256)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
{code}

The eventLog will be available normally if I restart the SparkThriftServer, I suggest that the EventLoggingListener's dfs writer and hadoopDataStream should reconnect after all datanodes stop and then start later。


{code:java}
  /** Log the event as JSON. */
  private def logEvent(event: SparkListenerEvent, flushLogger: Boolean = false) {
    val eventJson = JsonProtocol.sparkEventToJson(event)
    // scalastyle:off println
    writer.foreach(_.println(compact(render(eventJson))))
    // scalastyle:on println
    if (flushLogger) {
      writer.foreach(_.flush())
      hadoopDataStream.foreach(_.hflush())
    }
    if (testing) {
      loggedEvents += eventJson
    }
  }
{code}
Bug: StatCounter should use mergeStats when merging with self.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/21 22:00
Affects Version/s: 3.3.0

{{StatCounter}} should use {{mergeStats}} instead of {{merge}} when merging with {{self}}.

This is a long standing bug but usually this bug won't be hit unless users explicitly use {{mergeStats}} with {{self}}.
Bug: Flaky Test: DDLParserSuite.create view -- basic
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/21 18:34
Affects Version/s: 3.3.0

- https://github.com/apache/spark/runs/4188475706
- https://github.com/apache/spark/runs/4185560718
- https://github.com/apache/spark/runs/4174326895
- https://github.com/apache/spark/runs/4197748386

 

!Screen Shot 2021-11-12 at 10.34.32 AM.png!
Bug:  Resolved attribute(s) region_id#7993 missing from lm_tam_code#2666,ld_month#2697,geo_sub_district_id#2677 when doing left join
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/21 06:42
Affects Version/s: 2.4.0

For this Code i m getting error 

var df = totalTimeFootfallDf.join(dwellTimeDf,Seq(groupByKeys: _*),"left_outer")


org.apache.spark.sql.AnalysisException: Resolved attribute(s) region_id#7993 missing from lm_tam_code#2666,ld_month#2697,geo_sub_district_id#2677,lm_subldm_id#2652,geo_province_name_th#2689,lm_amp_code#2665,lm_cat_name_t#2675,lm_prov_code#2664,lm_sub_name_t#2672,lm_name_branch#2694,region_id#422,lm_update_dttm#2669,mkt_region_name_en#2692,lm_radius#369,geo_province_id#2679,lm_name#2655,geo_province_name_en#2684,lm_name_fld_t#2656,geo_region_name_en#2686,geo_district_name_th#2688,geo_sub_region_id#2680,ld_day#2698,lm_master#2653,lm_location_t#2660,lm_zone#2668,lm_latitude#2671,lm_branch_t#2658,geo_sub_district_name_th#2687,lm_cat_id#2674,geo_district_id#2678,lm_symbol#2649,lm_id#2648,lm_name_t#2654,lm_cat_name#2676,geo_district_name_en#2683,lm_name_fld#2657,lm_master_branch#2663,lm_longitude#2670,lm_branch#2659,mkt_sub_region_id#2690,lm_sub_cat_id#2650,geo_sub_region_name_en#2685,lm_sub_cat_name#316,mkt_sub_region_name_en#2691,grid_id#2695,lm_floor#2667,lm_location#2661,geo_sub_district_name_en#2682,lm_master_branch_t#2662,data_date#2693,lm_ldmtag#2651,ld_year#2696 in operator !Project [lm_id#2648, lm_symbol#2649, lm_sub_cat_id#2650, lm_ldmtag#2651, lm_subldm_id#2652, lm_master#2653, lm_name_t#2654, lm_name#2655, lm_name_fld_t#2656, lm_name_fld#2657, lm_branch_t#2658, lm_branch#2659, lm_location_t#2660, lm_location#2661, lm_master_branch_t#2662, lm_master_branch#2663, lm_prov_code#2664, lm_amp_code#2665, lm_tam_code#2666, lm_floor#2667, lm_zone#2668, lm_update_dttm#2669, lm_longitude#2670, lm_latitude#2671, ... 28 more fields]. Attribute(s) with the same name appear in the operation: region_id. Please check if the right attribute(s) are used.;;
Join LeftOuter
Bug: Deflake TPC-DS build 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/21 02:45
Affects Version/s: 3.3.0

TPC-DS is being flaky appearntly because of the lack of memories in GitHub Actions mahcines ,e.g.) https://github.com/apache/spark/runs/4239133536?check_suite_focus=true
Bug: Silence the `index_col` advice in `to_spark()` for internal usage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/21 05:49
Affects Version/s: 3.3.0

The advice warning for `to_spark()` - when the `index_col` parameter is not specified - issuing too much message e.g. when user runs the plotting functions, so we want to silence the warning message when it's used as an internal purpose.
Bug: pyspark.pandas.read_csv() should support latin-1 encoding
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/21 18:46
Affects Version/s: 3.2.0

{{In regular pandas, you can say read_csv(encoding='latin-1'). This encoding is not recognized in pyspark.pandas. You have to use Windows-1252 instead, which is almost the same but not identical. }}
Bug:  test_reuse_worker_of_parallelize_range is flaky
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/21 06:22
Affects Version/s: 3.3.0

 
{code:java}
ERROR [2.132s]: test_reuse_worker_of_parallelize_range (pyspark.tests.test_worker.WorkerReuseTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/tests/test_worker.py", line 195, in test_reuse_worker_of_parallelize_range
    self.assertTrue(pid in previous_pids)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 12 tests in 22.589s
{code}
 

 

[1] https://github.com/apache/spark/runs/1182154542?check_suite_focus=true
[2] https://github.com/apache/spark/pull/33657#issuecomment-893969310
[3] https://github.com/Yikun/spark/runs/4362783540?check_suite_focus=true
Bug: Pin `docutils` in branch-3.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/21 21:50
Affects Version/s: 3.1.3

`docutils` 0.18 is released October 26 and causes Python linter failure in branch-3.1.
- https://pypi.org/project/docutils/#history
- https://github.com/apache/spark/commits/branch-3.1

{code}
Exception occurred:
  File "/Users/dongjoon/.pyenv/versions/3.10.0/lib/python3.10/site-packages/docutils/writers/html5_polyglot/__init__.py", line 445, in section_title_tags
    if (ids and self.settings.section_self_link
AttributeError: 'Values' object has no attribute 'section_self_link'
The full traceback has been saved in /var/folders/mq/c32xpgtj4tj19vt8b10wp8rc0000gn/T/sphinx-err-2h05ytx2.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [html] Error 2
{code}
Bug: illegal reflective access operation has occurred; Please consider reporting this to the maintainers
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 11/Nov/21 21:28
Affects Version/s: 3.1.2

```

   spark = SparkSession\
                .builder\
                .appName("TestEstimatedScalingFactors")\
                .getOrCreate()

```

generates the following warning

```

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/xxx/googleUCSC/kimLab/extraCellularRNA/terra/deseq/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)

WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

21/11/11 12:51:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

```

I am using pyspark spark-3.1.2-bin-hadoop3.2 on a MacBook pro running mac OS 11.6

 

My small unit test see to work okay how ever It fails when I try and run on 3.2.0

 

I

 

Any idea how I track down this issue? Kind regards

 

Andy
Bug: Silence the to_pandas() advice log for internal usage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/21 03:44
Affects Version/s: 3.3.0

Raised from comment [https://github.com/apache/spark/pull/34389#discussion_r741733023].

The advice warning for pandas API on Spark for to_pandas() issuing too much message e.g. when user runs the plotting functions, so we want to silence the warning message when it's used as an internal purpose.
Bug: Add PandasAPIOnSparkAdviceWarning class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/21 00:35
Affects Version/s: 3.3.0

Raised from comment [https://github.com/apache/spark/pull/34389#discussion_r741733023].

The advice warning for pandas API on Spark for expensive APIs ([https://github.com/apache/spark/pull/34389#discussion_r741733023)|https://github.com/apache/spark/pull/34389#discussion_r741733023).] now issuing too much warning message, so it might be good to have pandas-on-Spark specific warning class so that users can manually turn it off by using warning.simplefilter.
Bug: Add the TO_BINARY() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/21 13:27
Affects Version/s: 3.3.0

to_binary(expr, fmt) is a common function available in many other systems to provide a unified entry for string to binary data conversion, where fmt can be utf8, base64, hex and base2 (or whatever the reverse operation to_char()supports).

[https://docs.aws.amazon.com/redshift/latest/dg/r_TO_VARBYTE.html]

[https://docs.snowflake.com/en/sql-reference/functions/to_binary.html]

[https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#format_string_as_bytes]

[https://docs.teradata.com/r/kmuOwjp1zEYg98JsB8fu_A/etRo5aTAY9n5fUPjxSEynw]

Related Spark functions: unbase64, unhex
Bug: PYSPARK Arrow 3.2.0 docs link invalid
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 17:34
Affects Version/s: 3.2.0

[http://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html]

links to:

[https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html]

which links to:

[https://spark.apache.org/docs/latest/api/python/sql/arrow_pandas.rst]

But that is an invalid link.

I assume its supposed to point to:

https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html
Bug: 100% CPU usage on Spark Thrift Server.
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/21 10:33
Affects Version/s: 3.1.2

We are trying to use Spark thrift server as a distributed sql query engine, the queries work when the resident memory occupied by Spark thrift server identified through HTOP is comparatively less than the driver memory. The same queries result in 100% cpu usage when the resident memory occupied by spark thrift server is greater than the configured driver memory and keeps running at 100% cpu usage. I am using incremental collect as false, as i need faster responses for exploratory queries. I am trying to understand the following points
 * Why isn't spark thrift server releasing back the memory, when there are no queries. 
 * What is causing spark thrift server to go into 100% cpu usage on all the cores, when spark thrift server's memory is greater than the driver memory (by 10% usually) and why are queries just stuck.
Bug: Support ANSI intervals in PySpark
Status: Resolved | Priority: Major | Resolution: Done
Created: 11/Nov/21 03:24
Affects Version/s: 3.3.0

This JIRA targets to implement ANSI interval types in PySpark:
- https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DayTimeIntervalType.scala
- https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/YearMonthIntervalType.scala