av9ash/GitBugs huggingface dataset:
Bug: The position of unresolved identifier for DDL commands should be respected..
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/20 06:21
Affects Version/s: 3.2.0

Currently, there are many DDL commands where the position of the unresolved identifiers are incorrect:

{code:java}
scala> sql("DESCRIBE TABLE abc")
org.apache.spark.sql.AnalysisException: Table or view not found: abc; line 1 pos 0;
{code}

Note that the pos should be 15 in this case.
Bug: When HiveDDL, we need check avro schema too like parquet & orc
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/20 03:40
Affects Version/s: 3.2.0

{code:java}
Caused by: org.apache.avro.SchemaParseException: Illegal initial character: (IF((1 = 1), 1, 0))
	at org.apache.avro.Schema.validateName(Schema.java:1147)
	at org.apache.avro.Schema.access$200(Schema.java:81)
	at org.apache.avro.Schema$Field.<init>(Schema.java:403)
	at org.apache.avro.Schema$Field.<init>(Schema.java:396)
	at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
	at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:170)
	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:114)
	at org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:121)
	at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:104)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)

{code}
Bug: InsertIntoDir failed since query column name contains ',' cause column type and column names size not equal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/20 04:12
Affects Version/s: 2.4.0

 

After hive-2.3 we will set COLUMN_NAME_DELIMITER to special char when col name cntains ','since column list and column types in serde is splited by COLUMN_NAME_DELIMITER.
 In spark-2.4.0 + hive-1.2.1 we will failed when INSERT OVERWRITE DIR when query result schema columan name contains ','as
{code:java}
 org.apache.hadoop.hive.serde2.SerDeException: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe: columns has 14 elements while columns.types has 11 elements! at org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters.extractColumnInfo(LazySerDeParameters.java:146) at org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters.<init>(LazySerDeParameters.java:85) at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initialize(LazySimpleSerDe.java:125) at org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:119) at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:103) at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120) at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:287) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:219) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:218) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$12.apply(Executor.scala:461) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:467) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){code}

 Since this problem has been solved by [https://github.com/apache/hive/blob/6f4c35c9e904d226451c465effdc5bfd31d395a0/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java#L1044-L1075]
 But I think we can do this in Spark side to make all version work well.
Bug: Upgrade built-in Hive to 2.3.8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 00:14
Affects Version/s: 3.2.0

Hive 2.3.8 changes:
 HIVE-19662: Upgrade Avro to 1.8.2
 HIVE-24324: Remove deprecated API usage from Avro
 HIVE-23980: Shade Guava from hive-exec in Hive 2.3
 HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
 HIVE-24512: Exclude calcite in packaging.
 HIVE-22708: Fix for HttpTransport to replace String.equals
Bug: Change datatype mapping in JDBC mssqldialect: DATETIME to DATETIME2
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Dec/20 19:04
Affects Version/s: 2.4.0

*datetime v/s datetime2*

Spark datetime type is [timestamptype|https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspark.apache.org%2Fdocs%2Flatest%2Fapi%2Fjava%2Forg%2Fapache%2Fspark%2Fsql%2Ftypes%2FTimestampType.html&data=04%7C01%7Cluxu1%40microsoft.com%7C39803a0f635646dadd6b08d89010896a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637417747986187437%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qPPJve%2FGAPeIp%2BI2hjB%2BqoGGN%2FcJQe6CIDjlEdUyASo%3D&reserved=0]. This supports a microsecond resolution.

 

Sql supports 2 date time types

o *datetime* can support only milli seconds resolution (0 to 999).

o *datetime2* is extension of datetime , is compatible with datetime and supports 0 to 9999999 sub second resolution.

Currently [MsSQLServerDialect|https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fblob%2Fbfb257f078854ad587a9e2bfe548cdb7bf8786d4%2Fsql%2Fcore%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fjdbc%2FMsSqlServerDialect.scala&data=04%7C01%7Cluxu1%40microsoft.com%7C39803a0f635646dadd6b08d89010896a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637417747986197428%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PMT9rA08NJRN0kwHy2ERaloOaDRB6ZsBBd70MZXl%2Bv4%3D&reserved=0] maps timestamptype to datetime. This implies results in errors when writing

*+Current+*
{code:java}
override def getJDBCType(dt: DataType): Option[JdbcType] = dt match { case TimestampType => Some(JdbcType("DATETIME", java.sql.Types.TIMESTAMP)) .. }

{code}
 

*+Proposal+*  
{code:java}
override def getJDBCType(dt: DataType): Option[JdbcType] = dt match { case TimestampType => Some(JdbcType("DATETIME2", java.sql.Types.TIMESTAMP)).. }
{code}
Bug: Add metrics for better tracking of dynamic allocation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/20 18:56
Affects Version/s: 3.2.0

We should add metrics to track the following:

1- Graceful decommissions & DA scheduled deletes

2- Jobs resubmitted

3- Fetch failures

4- Unexpected (e.g. non-Spark triggered) executor removals.
Bug: Spark-submit with --package deequ doesn't pull all jars
Status: Open | Priority: Major | Resolution: 
Created: 29/Dec/20 04:48
Affects Version/s: 2.4.7

This issue was marked as solved SPARK-24074; however, another [~hyukjin.kwon] pointed out in the comments that version 2.4x was experiencing this same problem when using Amazon Deequ.

This problem exist in 2.3.x ecosystem as well for Deequ.
Bug: Fix CBOs cost function 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/20 17:58
Affects Version/s: 2.4.7

The parameter spark.sql.cbo.joinReorder.card.weight is decumented as:
{code:title=spark.sql.cbo.joinReorder.card.weight}
The weight of cardinality (number of rows) for plan cost comparison in join reorder: rows * weight + size * (1 - weight).
{code}

But in the implementation the formula is a bit different:
{code:title=Current implementation}
    def betterThan(other: JoinPlan, conf: SQLConf): Boolean = {
      if (other.planCost.card == 0 || other.planCost.size == 0) {
        false
      } else {
        val relativeRows = BigDecimal(this.planCost.card) / BigDecimal(other.planCost.card)
        val relativeSize = BigDecimal(this.planCost.size) / BigDecimal(other.planCost.size)
        relativeRows * conf.joinReorderCardWeight +
          relativeSize * (1 - conf.joinReorderCardWeight) < 1
      }
    }
{code}

This change has an unfortunate consequence: 
given two plans A and B, both A betterThan B and B betterThan A might give the same results. This happes when one has many rows with small sizes and other has few rows with large sizes.

A example values, that have this fenomen with the default weight value (0.7):
A.card = 500, B.card = 300
A.size = 30, B.size = 80
Both A betterThan B and B betterThan A would have score above 1 and would return false.

A new implementation is proposed, that matches the documentation:
{code:title=Proposed implementation}
    def betterThan(other: JoinPlan, conf: SQLConf): Boolean = {
      val oldCost = BigDecimal(this.planCost.card) * conf.joinReorderCardWeight +
        BigDecimal(this.planCost.size) * (1 - conf.joinReorderCardWeight)
      val newCost = BigDecimal(other.planCost.card) * conf.joinReorderCardWeight +
        BigDecimal(other.planCost.size) * (1 - conf.joinReorderCardWeight)
      newCost < oldCost
    }
{code}
Bug: Avoid calling unregisterMapOutput when the map stage is being rerunning.
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Dec/20 23:36
Affects Version/s: 2.4.5

When a fetch failure happened, DAGScheduler will try to unregister the corresponding map output. The current logic has a race condition that the new map stage attempt is running while the current reduce stage attempt returns another fetch failure (note: the current reduce stage firstly returns a fetch failure to make the maps stage is rerunning, and then the rerunning map stage may return some mapstatus of the failed MapId before the current reduce stage returns another fetch failure at the same MapId, the current reduce is last attempt due to the new map stage is not yet completed). In this case, if the map output is always unregistered, it may actually unregister the map output from the new map stage attempt.
Bug: Upgrade Jackson to 2.11.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/20 09:37
Affects Version/s: 3.2.0

Upgrade Jackson to 2.11.4 to make it easy to upgrade Avro 1.10.1.

More details:
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.11

{noformat}
 com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.5 requires Jackson Databind version >= 2.10.0 and < 2.11.0
{noformat}
Bug: Push partial aggregates bellow exchanges
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Dec/20 18:50
Affects Version/s: 3.2.0

The `EnsureRequirements` rule inserts exchanges between partial and final aggregates to reduce the exchanged data amount. However, if the user manually partitions the data correctly, then partial aggregates will be after the exchange.

If possible we should push partial aggregates bellow the manually inserted exchanges to reduce the exchanged data amount.
Bug: Spark SQL unrecognized `db.tbl`
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 07/Dec/20 03:03
Affects Version/s: 2.4.2

When executing command like this:
{code:java}
create table `bigdata_db.test_spark_33686` (c1 int);
{code}
exception will thrown:
{code:java}
spark-sql> create table `bigdata_db.test_spark_33686` (c1 int);Error in query: `bigdata_db.test_spark_33686` is not a valid name for tables/databases. Valid names only contain alphabet characters, numbers and _.;
{code}
you must write command like this:
{code:java}
create table `bigdata_db`.`test_spark_33686` (c1 int);
{code}
i think we should support `db.tbl`
Bug: Respect case sensitivity in V1 SHOW PARTITIONS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/20 07:08
Affects Version/s: 2.4.7

SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config *spark.sql.caseSensitive* which is false by default, for instance:
{code:sql}
spark-sql> CREATE TABLE tbl1 (price int, qty int, year int, month int)
         > USING parquet
         > PARTITIONED BY (year, month);
spark-sql> INSERT INTO tbl1 PARTITION(year = 2015, month = 1) SELECT 1, 1;
spark-sql> SHOW PARTITIONS tbl1 PARTITION(YEAR = 2015, Month = 1);
Error in query: Non-partitioning column(s) [YEAR, Month] are specified for SHOW PARTITIONS;
{code}
Bug: Decommissioning Race Condition during Pod Snapshot
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 21:05
Affects Version/s: 3.1.0

Some version of Kubernetes may create a deletion timestamp field before changing the pod status to terminating, so a decommissioning node may have a deletion timestamp and a stage of running. Depending on when the K8s snapshot comes back this can cause a race condition with Spark believing the pod has been deleted before it has been.
Bug: UnionExec should require column ordering in RemoveRedundantProjects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 00:53
Affects Version/s: 3.1.0

UnionExec requires its children's columns to have the same order in order to merge the columns. Currently, the physical rule `RemoveRedundantProjects` can pass through the ordering requirements from its parent and incorrectly remove the necessary project nodes below a union operation.
Bug: JDBC datasource fails when reading spatial datatypes with the MS SQL driver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/20 17:40
Affects Version/s: 3.0.0

The MS SQL JDBC driver introduced support for spatial types since version 7.0. The JDBC data source lacks mappings for these types which results in an exception below. It seems that a mapping in MsSqlServerDialect.getCatalystType that maps -157 and -158 typecode to VARBINARY should address the issue.

 
{noformat}
java.sql.SQLException: Unrecognized SQL type -157
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getCatalystType(JdbcUtils.scala:251)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$getSchema$1(JdbcUtils.scala:321)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:321)
 at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:63)
 at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
 at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:364)
 at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:366)
 at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:355)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:355)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
 at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:381){noformat}
Bug: Duplicate field names causes wrong answers during aggregation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/20 20:49
Affects Version/s: 2.4.4

We saw this bug at Workday.

Duplicate field names for different fields can cause  org.apache.spark.sql.catalyst.expressions.RowBasedKeyValueBatch#allocate to return a fixed batch when it should have returned a variable batch leading to wrong results.

This example produces wrong results in the spark shell:

scala> sql("with T as (select id as a, -id as x from range(3)), U as (select id as b, cast(id as string) as x from range(3)) select T.x, U.x, min(a) as ma, min(b) as mb from T join U on a=b group by U.x, T.x").show
 
|*x*|*x*|*ma*|*mb*|
|-2|2|0|null|
|-1|1|null|1|
|0|0|0|0|

 instead of correct output : 
|*x*|*x*|*ma*|*mb*|
|0|0|0|0|
|-2|2|2|2|
|-1|1|1|1|

The issue can be solved by iterating over the fields themselves instead of field names.
Bug: Support save SHOW command result to table
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Dec/20 10:33
Affects Version/s: 3.1.0

The Scala interface support save SHOW command result to table:
{code:scala}
spark.sql("CREATE TABLE AS SHOW TABLES IN default").write.saveAsTable("t")
{code}

But SQL interface does not support it:
{code:sql}
spark-sql> create table t using parquet as SHOW TABLES IN default;
Error in query:
mismatched input 'SHOW' expecting {'(', 'FROM', 'MAP', 'REDUCE', 'SELECT', 'TABLE', 'VALUES', 'WITH'}(line 1, pos 32)

== SQL ==
create table t using parquet as SHOW TABLES IN default
--------------------------------^^^
{code}


It would be great if SQL interface can support it.
Bug: Fix flaky test "Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties."
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/20 07:23
Affects Version/s: 3.1.0

The test is flaking, with multiple flaked instances - the reason for the failure has been similar to:
{code:java}
  The code passed to eventually never returned normally. Attempted 109 times over 3.0079882413999997 minutes. Last failure message: Failure executing: GET at: https://192.168.39.167:8443/api/v1/namespaces/b37fc72a991b49baa68a2eaaa1516463/pods/spark-pi-97a9bc76308e7fe3-exec-1/log?pretty=false. Message: pods "spark-pi-97a9bc76308e7fe3-exec-1" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=null, kind=pods, name=spark-pi-97a9bc76308e7fe3-exec-1, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "spark-pi-97a9bc76308e7fe3-exec-1" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).. (KubernetesSuite.scala:402)
{code}

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36854/console
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36852/console
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36850/console
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36848/console

From the above failures, it seems, that executor finishes too quickly and is removed by spark before the test can complete. 

So, in order to mitigate this situation, one way is to turn on the flag

{code}
   "spark.kubernetes.executor.deleteOnTermination"
{code}
Bug:  Race condition in Spark k8s Pod lifecycle manager that leads to shutdowns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 16:57
Affects Version/s: 2.3.4

Watching a POD (ExecutorPodsWatchSnapshotSource) informs about single POD changes which could wrongfully lead to detecting of missing PODs (PODs known by scheduler backend but missing from POD snapshots) by the executor POD lifecycle manager.

A key indicator of this is seeing this log msg:

"The executor with ID [some_id] was not found in the cluster but we didn't get a reason why. Marking the executor as failed. The executor may have been deleted but the driver missed the deletion event."

So one of the problem is running the missing POD detection even when a single pod is changed without having a full consistent snapshot about all the PODs (see ExecutorPodsPollingSnapshotSource). The other could be a race between the executor POD lifecycle manager and the scheduler backend.
Bug: cache table not working for persisted view
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/20 09:54
Affects Version/s: 3.0.1

In `CacheManager`, tables (including views) are cached by its logical plan, and
use `QueryPlan.sameResult` to lookup the cache. But the PersistedView wraps
the child plan with a `View` which always lead false for `sameResult` check.
Bug: When refreshing cache, Spark should not use cached plan when recaching data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Dec/20 01:14
Affects Version/s: 3.1.0

Currently when cache is refreshed, e.g., via "REFRESH TABLE" command, Spark will call {{refreshTable}} method within {{CatalogImpl}}.

{code}
  override def refreshTable(tableName: String): Unit = {
    val tableIdent = sparkSession.sessionState.sqlParser.parseTableIdentifier(tableName)
    val tableMetadata = sessionCatalog.getTempViewOrPermanentTableMetadata(tableIdent)
    val table = sparkSession.table(tableIdent)

    if (tableMetadata.tableType == CatalogTableType.VIEW) {
      // Temp or persistent views: refresh (or invalidate) any metadata/data cached
      // in the plan recursively.
      table.queryExecution.analyzed.refresh()
    } else {
      // Non-temp tables: refresh the metadata cache.
      sessionCatalog.refreshTable(tableIdent)
    }

    // If this table is cached as an InMemoryRelation, drop the original
    // cached version and make the new version cached lazily.
    val cache = sparkSession.sharedState.cacheManager.lookupCachedData(table)

    // uncache the logical plan.
    // note this is a no-op for the table itself if it's not cached, but will invalidate all
    // caches referencing this table.
    sparkSession.sharedState.cacheManager.uncacheQuery(table, cascade = true)

    if (cache.nonEmpty) {
      // save the cache name and cache level for recreation
      val cacheName = cache.get.cachedRepresentation.cacheBuilder.tableName
      val cacheLevel = cache.get.cachedRepresentation.cacheBuilder.storageLevel

      // recache with the same name and cache level.
      sparkSession.sharedState.cacheManager.cacheQuery(table, cacheName, cacheLevel)
    }
  }
{code}

Note that the {{table}} is created before the table relation cache is cleared, and used later in {{cacheQuery}}. This is incorrect since it still refers cached table relation which could be stale.
Bug: String functions: Trim/Ltrim/Rtrim support byte array
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Dec/20 07:46
Affects Version/s: 3.2.0

String functions: Trim/Ltrim/Rtrim support byte arrays

The mainstream database support this feature show below:

Teradata
https://docs.teradata.com/r/756LNiPSFdY~4JcCCcR5Cw/vd81iWAoGj0cEeGBMAfF9A

Vertica
https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/String/BTRIM.htm?tocpath=SQL%20Reference%20Manual%7CSQL%20Functions%7CString%20Functions%7C_____5

Redshift
https://docs.aws.amazon.com/redshift/latest/dg/r_BTRIM.html

Postgresql
https://www.postgresql.org/docs/11/functions-binarystring.html
Bug: limit partition num to 1 when distributing by foldable expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/20 08:26
Affects Version/s: 3.2.0

It seems a very popular way that people use DISTRIBUTE BY clause with a literal to coalesce partition in the pure SQL data processing.

For example
```
insert into table src select * from values (1), (2), (3) t(a) distribute by 1
```

Users may want the final output to be one single data file, but if the reality is not always true. Spark will always create a file for partition 0 whether it contains data or not, so when the data all goes to a partition(IDX >0), there will be always 2 files there and the part-00000 is empty. On the other hand, a lot of empty tasks will be launched too, this is unnecessary.

When users repeat the insert statement daily, hourly, or minutely, it causes small file issues.

To avoid this, there are some options you can take.

1. user `distribute by null`, let the data go to the partition 0
2. set spark.sql.adaptive.enabled to true for Spark to automatically coalesce
3. using hints instead of `distribute by`
4. set spark.sql.shuffle.partitions to 1