av9ash/GitBugs huggingface dataset:
Bug: GROUP BY using literal map should not fail
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 09:23
Affects Version/s: 2.0.2

Apache Spark 2.x ~ 3.0.1 raise`RuntimeException` for the following queries.
*SQL*
{code}
CREATE TABLE t USING ORC AS SELECT map('k1', 'v1') m, 'k1' k
SELECT map('k1', 'v1')[k] FROM t GROUP BY 1
SELECT map('k1', 'v1')[k] FROM t GROUP BY map('k1', 'v1')[k]
SELECT map('k1', 'v1')[k] a FROM t GROUP BY a
{code}

*ERROR*
{code}
Caused by: java.lang.RuntimeException: Couldn't find k#3 in [keys: [k1], values: [v1][k#3]#6]
	at scala.sys.package$.error(package.scala:27)
	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:85)
	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:79)
	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
{code}

This is a regression from Apache Spark 1.6.x.
{code}
scala> sc.version
res1: String = 1.6.3

scala> sqlContext.sql("SELECT map('k1', 'v1')[k] FROM t GROUP BY map('k1', 'v1')[k]").show
+---+
|_c0|
+---+
| v1|
+---+
{code}
Bug: Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/20 00:33
Affects Version/s: 3.1.0

DiskBlockManager should be able to create the {{merge_manager}} directory, where the push-based merged shuffle files are written and also create sub-dirs under it. 

It should also be able to serve the local merged shuffle data/index/meta files.
Bug: Incorrect join results when joining twice with the same DF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 14:19
Affects Version/s: 3.0.0

{code:java}
val emp1 = Seq[TestData](
  TestData(1, "sales"),
  TestData(2, "personnel"),
  TestData(3, "develop"),
  TestData(4, "IT")).toDS()
val emp2 = Seq[TestData](
  TestData(1, "sales"),
  TestData(2, "personnel"),
  TestData(3, "develop")).toDS()
val emp3 = emp1.join(emp2, emp1("key") === emp2("key")).select(emp1("*"))
emp1.join(emp3, emp1.col("key") === emp3.col("key"), "left_outer").select(emp1.col("*"), emp3.col("key").as("e2")).show()

// wrong result
+---+---------+---+
|key|    value| e2|
+---+---------+---+
|  1|    sales|  1|
|  2|personnel|  2|
|  3|  develop|  3|
|  4|       IT|  4|
+---+---------+---+

{code}
Bug: Update  doc about grouping analytics
Status: Closed | Priority: Major | Resolution: Implemented
Created: 07/Nov/20 02:03
Affects Version/s: 3.1.0

Need to update doc  after all  done.

[https://github.com/apache/spark/pull/30212#discussion_r518745351]

[https://spark.apache.org/docs/3.0.1/sql-ref-syntax-qry-select-groupby.html]
Bug: Support composite columns in cube/rollup
Status: Closed | Priority: Major | Resolution: Incomplete
Created: 07/Nov/20 01:47
Affects Version/s: 3.1.0

Oracle:

[https://docs.oracle.com/cd/E11882_01/server.112/e25554/aggreg.htm#DWHSG8634]

!image-2020-11-07-09-48-17-739.png!

 

MySQL not support cube/rollup/grouping sets:

!image-2020-11-07-09-49-18-161.png!

 

PostgresSQL support composite column in CUBE/ROLLUP:

!image-2020-11-07-09-49-59-190.png!
Bug: Group exception messages in execution/datasources/v2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 07:59
Affects Version/s: 3.2.0

'/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2'
|| Filename                         ||   Count ||
| AlterTableExec.scala             |       1 |
| CreateNamespaceExec.scala        |       1 |
| CreateTableExec.scala            |       1 |
| DataSourceRDD.scala              |       2 |
| DataSourceV2Strategy.scala       |       9 |
| DropNamespaceExec.scala          |       2 |
| DropTableExec.scala              |       1 |
| EmptyPartitionReader.scala       |       1 |
| FileDataSourceV2.scala           |       1 |
| FilePartitionReader.scala        |       2 |
| FilePartitionReaderFactory.scala |       1 |
| ReplaceTableExec.scala           |       3 |
| TableCapabilityCheck.scala       |       2 |
| V1FallbackWriters.scala          |       1 |
| V2SessionCatalog.scala           |      14 |
| WriteToDataSourceV2Exec.scala    |      10 |

'/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc'
|| Filename               ||   Count ||
| JDBCTableCatalog.scala |       3 |

'/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2'
|| Filename                    ||   Count ||
| DataSourceV2Implicits.scala |       3 |
Bug: Group exception messages in execution/datasources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 08:21
Affects Version/s: 3.2.0

'/core/src/main/scala/org/apache/spark/sql/execution/datasources'
|| Filename                                ||   Count ||
| DataSource.scala                        |       9 |
| DataSourceStrategy.scala                |       1 |
| DataSourceUtils.scala                   |       2 |
| FileFormat.scala                        |       1 |
| FileFormatWriter.scala                  |       3 |
| FileScanRDD.scala                       |       2 |
| InsertIntoHadoopFsRelationCommand.scala |       2 |
| PartitioningAwareFileIndex.scala        |       1 |
| PartitioningUtils.scala                 |       3 |
| RecordReaderIterator.scala              |       1 |
| rules.scala                             |       4 |


'/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile'
|| Filename               ||   Count ||
| BinaryFileFormat.scala |       2 |


'/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc'
|| Filename          ||   Count ||
| JDBCOptions.scala |       2 |
| JdbcUtils.scala   |       6 |


'/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc'
|| Filename              ||   Count ||
| OrcDeserializer.scala |       1 |
| OrcFilters.scala      |       1 |
| OrcSerializer.scala   |       1 |
| OrcUtils.scala        |       2 |


'/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet'
|| Filename                     ||   Count ||
| ParquetFileFormat.scala      |       2 |
| ParquetReadSupport.scala     |       1 |
| ParquetSchemaConverter.scala |       6 |
Bug: Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 06:15
Affects Version/s: 3.1.0

Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid, when ANSI mode is enable. This patch should update GetTimeStamp, UnixTimeStamp, ToUnixTimeStamp and Cast
Bug: Incorrect value when inserting into date type partition table with date type value
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/20 03:21
Affects Version/s: 3.1.0

{code:java}
create table test_part (name STRING) partitioned by (part date) STORED AS PARQUET;
insert into test_part partition(part = date '2019-01-02') values('a');
select * from test_part;
{code}


{noformat}
spark-sql> select * from test_part;
a	NULL
{noformat}
Bug: calling system.exit(-1) from a future does not result in the driver reflecting the fact that  the spark app failed.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 19/Nov/20 14:25
Affects Version/s: 2.4.0

Hi Guys,

when calling system. exit(-1) from within a code block defined for a future observed that the vm does not exit immediately as you would expect a java app to do. in fact it seems to exit the future thread and the diver program carries on as normal. the status of the app when it exits is also misleading. It says FINISHED instead of FAILED. I have attached code sample see below) that replicate this issue. is this a spark bug or a "feature" :)  any thoughts?

Note: I have been able to work around this by not calling system .exit(-1) from within a future but this was definitely a gotcha that I thought worthwhile raising..  
{code:java}
// code placeholder
package com.spark.example

import com.typesafe.config.Config
import com.typesafe.scalalogging.Logger
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.DurationInt
import scala.concurrent.{Await, Future}
import scala.util.{Failure, Success, Try}

object SystemExitTestApp extends App {
  val appName = "NumberCountSparkApp"

  val logger = Logger(getClass)
  val sparkMaster = "spark://adl-matspm01:7077"

  val ss = SparkSession.builder()
    .master(sparkMaster)
    .appName(appName).getOrCreate()


  logger.info(s"Starting test. thread id ${Thread.currentThread().getId}")

  val data = Seq(1, 2, 3, 4)
  val rdd: RDD[Int] = ss.sparkContext.parallelize(data)
  val future = Future {
    logger.info(s"inside future thread id ${Thread.currentThread().getId}")
    Try {
      throw new Exception("My dummy exception")
      rdd.count()
    } match {
      case Success(_) => {
        (true, 1)
      }
      case Failure(exception) => {
        logger.info(s"inside future. Exception ${exception} thread id ${Thread.currentThread().getId}")
        (false, 0)
      }
    }
  }
  var result: Boolean = true
  future onComplete {
    case Success(x) => {
      logger.info(s"success ${x}")
      if (!x._1) {
        logger.info(s"exiting in the success block of on complete. thread id ${Thread.currentThread().getId}")
        result = x._1
        ss.close()
        sys.exit(-1)
      } else {
        logger.info(s"not exiting in the success block of on complete. thread id ${Thread.currentThread().getId}")
        result = x._1
      }
    }
    case Failure(exception) => {
      logger.info(s"exiting in the failure block of on complete as exception occurred ${exception}. thread id ${Thread.currentThread().getId}")
      result = false
      sys.exit(-1)
    }
  }
  Await.result(future, 3 hours)
  logger.info(s"should not see this really if we do lets use the result outside. thread id ${Thread.currentThread().getId}")
  if (!result) {
    logger.info(s" inside main block should not see this really. thread id ${Thread.currentThread().getId}")
    //sys.exit(-1)
  }


}

{code}
Bug: The application log in the Spark history server contains sensitive attributes such as password that should be redated instead of plain text
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 15:54
Affects Version/s: 3.0.1

We found the secure attributes in SparkListenerJobStart and SparkListenerStageSubmitted events would not been redated, resulting in sensitive attributes can be viewd directly.
Bug: BroadCastJoin failure when keys on join side has cast from DateTyte to String
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 11/Nov/20 08:39
Affects Version/s: 3.0.1

when use spark as below : 

spark.sql(
 """ create table table1(a1 INT , a2 STRING)
 | using parquet
 |
 |""".stripMargin).show()

spark.sql(
 """ create table table2(b1 INT , b2 STRING)
 | using parquet
 |
 |""".stripMargin).show()

spark.sql(
 """ select /*+ BROADCAST(a) */ * from table1 a join table2 b
 | on cast(to_date(a.a2) as string) = b.b2
 |
 |""".stripMargin).show()

Exception following will be thrown : 

 java.util.NoSuchElementException: None.getjava.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:529) at scala.None$.get(Option.scala:527) at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:56) at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:56) at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId$lzycompute(Cast.scala:253) at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId(Cast.scala:253) at org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter$lzycompute(Cast.scala:287)
Bug: Group exception messages in catalyst/expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 18:30
Affects Version/s: 3.2.0


'/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions'
|| Filename                          ||   Count ||
| Cast.scala                        |      18 |
| ExprUtils.scala                   |       2 |
| Expression.scala                  |       8 |
| InterpretedUnsafeProjection.scala |       1 |
| ScalaUDF.scala                    |       2 |
| SelectedField.scala               |       3 |
| SubExprEvaluationRuntime.scala    |       1 |
| arithmetic.scala                  |       8 |
| collectionOperations.scala        |       4 |
| complexTypeExtractors.scala       |       3 |
| csvExpressions.scala              |       3 |
| datetimeExpressions.scala         |       4 |
| decimalExpressions.scala          |       2 |
| generators.scala                  |       2 |
| higherOrderFunctions.scala        |       6 |
| jsonExpressions.scala             |       2 |
| literals.scala                    |       3 |
| misc.scala                        |       2 |
| namedExpressions.scala            |       1 |
| ordering.scala                    |       1 |
| package.scala                     |       1 |
| regexpExpressions.scala           |       1 |
| stringExpressions.scala           |       1 |
| windowExpressions.scala           |       5 |


'/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate'
|| Filename                    ||   Count ||
| ApproximatePercentile.scala |       2 |
| HyperLogLogPlusPlus.scala   |       1 |
| Percentile.scala            |       1 |
| interfaces.scala            |       2 |


'/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen'
|| Filename            ||   Count ||
| CodeGenerator.scala |       5 |
| javaCode.scala      |       1 |


'/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects'
|| Filename      ||   Count ||
| objects.scala |      12 |
Bug: Group exception messages in catalyst/catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 18:47
Affects Version/s: 3.2.0

Group all exception messages in sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog.
||Filename||Count||
|ExternalCatalog.scala|4|
|GlobalTempViewManager.scala|1|
|InMemoryCatalog.scala|18|
|SessionCatalog.scala|17|
|functionResources.scala|1|
|interface.scala|4|
Bug: Group exception messages in catalyst/analysis
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 07:56
Affects Version/s: 3.2.0

'/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis'
|| Filename               ||   Count ||
| Analyzer.scala         |       1 |
| CheckAnalysis.scala    |       1 |
| FunctionRegistry.scala |       5 |
| ResolveCatalogs.scala  |       1 |
| ResolveHints.scala     |       1 |
| package.scala          |       2 |
| unresolved.scala       |      43 |

'/core/src/main/scala/org/apache/spark/sql/catalyst/analysis'
|| Filename                    ||   Count ||
| ResolveSessionCatalog.scala |      12 |
Bug: Remove the option of "sharesHadoopClasses" in Hive IsolatedClientLoader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Nov/20 22:29
Affects Version/s: 3.0.1

Currently, when initializing {{IsolatedClientLoader}}, ppl can specify to either share Hadoop classes from Spark or not. In the latter case it's supposed to only loads the Hadoop classes from the Hive jars themselves.

However this feature is currently used in two cases: 1) unit tests, 2) when the Hadoop version defined in Maven can not be found when {{spark.sql.hive.metastore.jars == "maven"}}. Also when {{sharesHadoopClasses}} is false, it isn't really only using Hadoop classes from Hive jars: Spark also download {{hadoop-client}} jar and put it together with the Hive jars, and the Hadoop version used by {{hadoop-client}} is the same version used by Spark itself. This could potentially cause issues because we are mixing two versions of Hadoop jars in the classpath.
Bug: Resolve SQL query reference a column by an alias
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 10/Nov/20 10:42
Affects Version/s: 3.1.0

This pr add support resolve SQL query reference a column by an alias, for example:
 ```sql
 select id + 1 as new_id, new_id + 1 as new_new_id from range(5);
 ```

Teradata and snowflake support this feature: [https://docs.teradata.com/reader/e79ET77~NzPDz~Ykinj44w/MKSYuTyx2UJWXzdHJf3~sQ]

[https://www.mssqltips.com/sqlservertutorial/9292/snowflake-regular-expression-alias-and-ilike]
Bug: NULL is recognized as the "null" string in partition specs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/20 19:27
Affects Version/s: 3.1.0

For example:
{code:sql}
spark-sql> CREATE TABLE tbl5 (col1 INT, p1 STRING) USING PARQUET PARTITIONED BY (p1);
spark-sql> INSERT INTO TABLE tbl5 PARTITION (p1 = null) SELECT 0;
spark-sql> SELECT isnull(p1) FROM tbl5;
false
{code}

The *p1 = null* is not recognized as a partition with NULL value.
Bug: Improve performance of ML ALS recommendForAll by GEMV
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/20 10:55
Affects Version/s: 3.1.0

There were a lot of works on improving ALS's {{recommendForAll}}

For now, I found that it maybe futhermore optimized by

1, using GEMV;

2, use guava.ordering instead of BoundedPriorityQueue;
Bug: Support REGEXP_LIKE for consistent with mainstream databases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 07:33
Affects Version/s: 3.1.0

There are a lot of mainstream databases support regex function REGEXP_LIKE.
Currently, Spark supports RLike and we just need add a new alias REGEXP_LIKE for it.
*Oracle*:https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/Pattern-matching-Conditions.html#GUID-D2124F3A-C6E4-4CCA-A40E-2FFCABFD8E19
*Presto*:https://prestodb.io/docs/current/functions/regexp.html
*Vertica*:https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/RegularExpressions/REGEXP_LIKE.htm?tocpath=SQL%20Reference%20Manual%7CSQL%20Functions%7CRegular%20Expression%20Functions%7C_____5
*Snowflake*:https://docs.snowflake.com/en/sql-reference/functions/regexp_like.html
Bug: Add dataSourceRewriteRules batch to Optimizer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 17:59
Affects Version/s: 3.1.0

Right now, we have a special place in the optimizer where we run rules for constructing v2 scans. As time shows, we need more rewrite rules for v2 tables and not all of them related to reads. One option is to rename the current batch into something more generic but it would require changing quite some places. Moreover, the batch contains some non-V2 logic as well. That's why it seems better to introduce a new batch and use it for all data source v2 rewrites and beyond.
Bug: CREATE TABLE LIKE should resolve hive serde correctly like CREATE TABLE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 01:25
Affects Version/s: 2.4.0

Currently there are several inconsistency:
 # CREATE TABLE LIKE does not validate the user-specified hive serde. e.g., STORED AS PARQUET can't be used with ROW FORMAT SERDE.
 # CREATE TABLE LIKE requires STORED AS and ROW FORMAT SERDE to be specified together, which is not necessary.
 # CREATE TABLE LIKE does not respect the default hive serde.
Bug: Unify v1 and v2 ALTER TABLE .. ADD PARTITION tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 11:29
Affects Version/s: 3.1.0

Extract ALTER TABLE .. ADD PARTITION tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Extend the function of decode so as consistent with mainstream databases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 07:29
Affects Version/s: 3.1.0

In Spark, decode(bin, charset) - Decodes the first argument using the second argument character set.

Unfortunately this is NOT what any other SQL vendor understands DECODE to do.
DECODE generally is a short hand for a simple case expression:


{code:java}
SELECT DECODE(c1, 1, 'Hello', 2, 'World', '!') FROM (VALUES (1), (2), (3)) AS T(c1)
=> 
(Hello),
(World)
(!)
{code}
Bug: Partition predicate pushdown into Hive metastore support cast string type to date type
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Nov/20 09:46
Affects Version/s: 3.1.0

{code:scala}
spark.sql("create table t1(id string) partitioned by (part string) stored as parquet")
spark.sql("insert into t1 values('1', '2019-01-01')")
spark.sql("insert into t1 values('2', '2019-01-02')")
spark.sql("select * from t1 where  part = date '2019-01-01' ").show
{code}

We can pushdown {{cast(part as date) = date '2019-01-01' }} to Hive metastore.
Bug: Upgrade hive-service-rpc to 3.1.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 02:56
Affects Version/s: 3.1.0

We supported Hive metastore are 0.12.0 through 3.1.2. but we supported hive-jdbc are 0.12.0 through 2.3.7. It will throw TProtocolException if we use hive-jdbc 3.x:

{noformat}
[root@spark-3267648 apache-hive-3.1.2-bin]# bin/beeline -u jdbc:hive2://localhost:10000/default
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Spark SQL (version 3.1.0-SNAPSHOT)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.2 by Apache Hive
0: jdbc:hive2://localhost:10000/default> create table t1(id int) using parquet;
Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable.
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
{noformat}

{noformat}
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:234)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
{noformat}

We can upgrade hive-service-rpc to 3.1.2 to fix this issue.
Bug: [CORE] Add config spark.executor.coresOverhead
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 13/Nov/20 17:16
Affects Version/s: 3.0.1

Add config spark.executor.coresOverhead to request extra cores per executor. This config will be helpful in below cases:

Suppose for physical machines or vm, the memory/cpu ratio is 3GB/core. But we run spark job, we want to have 6GB per task. If we request resource in such way, there will be resource waste.
If we request extra cores without affecting cores per executor for task allocation, extra cores won't be wasted.