av9ash/GitBugs huggingface dataset:
Bug: Restore previous behaviour of randomSplit from spark-2.4.7 in spark-3
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 18/Mar/21 19:48
Affects Version/s: 3.0.1

Hi, 

Please consider a following difference of `randomSplit` method even despite of the same seed.

 
{code:java}
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.7
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_282)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val Array(f, s) =  Seq(1,2,3,4,5,6,7,8,9,10).toDF.randomSplit(Array(0.3, 0.7), 42)
f: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
s: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
scala> f.show
+-----+
|value|
+-----+
|    4|
+-----+
scala> s.show
+-----+
|value|
+-----+
|    1|
|    2|
|    3|
|    5|
|    6|
|    7|
|    8|
|    9|
|   10|
+-----+
{code}
while as on spark-3
{code:java}
scala> val Array(f, s) =  Seq(1,2,3,4,5,6,7,8,9,10).toDF.randomSplit(Array(0.3, 0.7), 42)
f: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
s: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
scala> f.show
+-----+
|value|
+-----+
|    5|
|   10|
+-----+
scala> s.show
+-----+
|value|
+-----+
|    1|
|    2|
|    3|
|    4|
|    6|
|    7|
|    8|
|    9|
+-----+
{code}
I guess that implementation of `sample` method changed.

Is it possible to restore previous behaviour?

Thanks in advance!
Bug: Introduce Jetty based construct for integration tests where HTTP(S) is used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/21 16:04
Affects Version/s: 3.2.0

This came up during https://github.com/apache/spark/pull/31877#discussion_r596831803.

Short summary: we have some tests where HTTP(S) is used to access files. The current solution uses github urls like "https://raw.githubusercontent.com/apache/spark/master/data/mllib/pagerank_data.txt".
This connects two Spark version in an unhealthy way like connecting the "master" branch which is moving part with the committed test code which is a non-moving (as it might be even released).
So this way a test running for an earlier version of Spark expects something (filename, content, path) from a the latter release and what is worse when the moving version is changed the earlier test will break. 

The idea is to introduce a method like:

{noformat}
withHttpServer(files) {
}
{noformat}

Which uses a Jetty ResourceHandler to serve the listed files (or directories / or just the root where it is started from) and stops the server in the finally.
Bug: Enable ANSI intervals in SQLQueryTestSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/21 11:17
Affects Version/s: 3.2.0

Remove the following code from SQLQueryTestSuite:
{code:java}
    localSparkSession.conf.set(SQLConf.LEGACY_INTERVAL_ENABLED.key, true)
{code}
and use the ANSI interval where it is possible.

Probably, this depends on casting intervals to strings.
Bug: Use better CBO cost function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/21 12:01
Affects Version/s: 3.2.0

In SPARK-33935 we changed the CBO cost function such that it would be symetric - A.betterThan(B) implies that !B.betterThan(A). Before both could have been true.

That change introduced a performance regression in some queries.
Bug: Add table function registry
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/21 19:04
Affects Version/s: 3.2.0

Currently, Spark resolves table-valued functions, for example, range, very differently from non-table valued functions. Also, there is no information displayed for table-valued functions. For example:
{code:scala}
scala> sql("describe function range").show(false)
+--------------------------+
|function_desc             |
+--------------------------+
|Function: range not found.|
+--------------------------+
{code}
To make the behavior consistent, we can add a table function registry and resolve the table-valued function from the catalog.

This also allows Spark to have more built-in table-valued functions in the future.
Bug: Refactor TreeNode's children handling methods into specialized traits
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/21 13:45
Affects Version/s: 3.1.1

Spark query plan node hierarchy has specialized traits (or abstract classes) for handling nodes with fixed number of children, for example UnaryExpression, UnaryNode and UnaryExec for representing an expression, a logical plan and a physical plan with only one child, respectively. This PR refactors the TreeNode hierarchy by extracting the children handling functionality into the following traits. UnaryExpression` and other similar classes now extend the corresponding new trait:

{{trait LeafLike[T <: TreeNode[T]] { self: TreeNode[T] =>}}
 {{  override final def children: Seq[T] = Nil}}
 {{}}}

{{trait UnaryLike[T <: TreeNode[T]] { self: TreeNode[T] =>}}
 {{  def child: T}}
 {{  @transient override final lazy val children: Seq[T] = child :: Nil}}
 {{}}}

{{trait BinaryLike[T <: TreeNode[T]] { self: TreeNode[T] =>}}
 {{  def left: T}}
 {{  def right: T}}
 {{  @transient override final lazy val children: Seq[T] = left :: right :: Nil}}
 {{}}}

{{trait TernaryLike[T <: TreeNode[T]] { self: TreeNode[T] =>}}
 {{  def first: T}}
 {{  def second: T}}
 {{  def third: T}}
 {{  @transient override final lazy val children: Seq[T] = first :: second :: third :: Nil}}
 {{}}}

 

This refactoring, which is part of a bigger effort to make tree transformations in Spark more efficient, has two benefits:
 * It moves the children handling to a single place, instead of being spread in specific subclasses, which will help the future optimizations for tree traversals.
 * It allows to mix in these traits with some concrete node types that could not extend the previous classes. For example, expressions with one child that extend AggregateFunction cannot extend UnaryExpression as AggregateFunction defines the foldable method final while UnaryExpression defines it as non final. With the new traits, we can directly extend the concrete class from UnaryLike in these cases. Classes with more specific child handling will make tree traversal methods faster.

In this PR we have also updated many concrete node types to extend these traits to benefit from more specific child handling.
Bug: Return UDT from Pandas UDF: @pandas_udf(UDT)
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Mar/21 02:51
Affects Version/s: 3.0.2

Focus on a simpler case first: `@pandas_udf(UDT)`.




Because Pandas UDF uses pyarrow to passing data, it does not currently support UserDefinedTypes, as what normal python udf does.

For example:


{code:python}
class BoxType(UserDefinedType):
    @classmethod
    def sqlType(cls) -> StructType:
        return StructType(
            fields=[
                StructField("xmin", DoubleType(), False),
                StructField("ymin", DoubleType(), False),
                StructField("xmax", DoubleType(), False),
                StructField("ymax", DoubleType(), False),
            ]
        )

@pandas_udf(
     returnType=StructType([StructField("boxes", ArrayType(Box()))]
)
def pandas_pf(s: pd.DataFrame) -> pd.DataFrame:
       yield s
{code}

The logs show
{code}
try:
                to_arrow_type(self._returnType_placeholder)
            except TypeError:
>               raise NotImplementedError(
                    "Invalid return type with scalar Pandas UDFs: %s is "
E                   NotImplementedError: Invalid return type with scalar Pandas UDFs: StructType(List(StructField(boxes,ArrayType(Box,true),true))) is not supported
{code}
Bug: RewriteDistinctAggregates can cause a bug if the aggregator does not ignore NULLs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/21 19:09
Affects Version/s: 2.4.8

{code:title=group-by.sql}
SELECT
    first(DISTINCT a), last(DISTINCT a),
    first(a), last(a),
    first(DISTINCT b), last(DISTINCT b),
    first(b), last(b)
FROM testData WHERE a IS NOT NULL AND b IS NOT NULL;{code}
{code:title=group-by.sql.out}
-- !query schema
struct<first(DISTINCT a):int,last(DISTINCT a):int,first(a):int,last(a):int,first(DISTINCT b):int,last(DISTINCT b):int,first(b):int,last(b):int>
-- !query output
NULL	1	1	3	1	NULL	1	2
{code}

The results should not be NULL, because NULL inputs are filtered out.
Bug: cannot resolve methods when & col in spark java
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 25/Mar/21 09:57
Affects Version/s: 3.1.1

Hello,

 

I am using Spark 3.1.1 with JAVA and the method when of Column and also col but when i compile it says cannot resolve method when & cannot resolve method col.

 

What is the matter ?
Bug: cannot resolve methods when & col in spark java
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Mar/21 09:58
Affects Version/s: 3.1.1

Hello,

 

I am using Spark 3.1.1 with JAVA and the method when of Column and also col but when i compile it says cannot resolve method when & cannot resolve method col.

 

What is the matter ?
Bug: Start spark shell to read file and report an error
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 31/Mar/21 02:36
Affects Version/s: 1.6.0

Start spark shell to read external file and report an error

 

Class org.apache.hadoop .fs.s3a.S3AFileSystem not found

 

Spark version 1.6 and Hadoop version 2.6.0 have copied aws-java-sdk-1.7.4.jar and hadoop-aws-2.6.0.jar to the Lib package under spark
Bug: Start spark shell to read file and report an error
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 31/Mar/21 02:37
Affects Version/s: 1.6.0

Start spark shell to read external file and report an error

Class org.apache.hadoop .fs.s3a.S3AFileSystem not found

Spark version 1.6 and Hadoop version 2.6.0 have copied aws-java-sdk-1.7.4.jar and hadoop-aws-2.6.0.jar to the Lib package under spark
Bug: ANSI mode: Allow casting complex types as string type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/21 15:12
Affects Version/s: 3.2.0

Currently, complex types are not allowed to cast as string type. This breaks the Dataset.show() API. E.g

{code:java}
scala> sql(“select array(1, 2, 2)“).show(false)
org.apache.spark.sql.AnalysisException: cannot resolve ‘CAST(`array(1, 2, 2)` AS STRING)’ due to data type mismatch:
 cannot cast array<int> to string with ANSI mode on.
{code}
Bug: Use 'io.connectionTimeout' as a hint instead of `spark.network.timeout`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/21 06:15
Affects Version/s: 2.4.7

Currently, when a connection for TransportClient is marked as idled and closed, we suggest users adjust `spark.network.timeout` for all transport modules. As a lot of timeout configs will fallback to the `spark.network.timeout`, this could be a piece of overkill advice, we should give a more targeted one with `spark.${moduleName}.io.connectionTimeout`
Bug: Push down limit through Project
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/21 16:15
Affects Version/s: 3.2.0

We will insert a Project if the Join output is not match the LocalLimit. We can not push down limit through join in this case. For example:
{code:scala}
spark.sql("create table t1(a int, b int, c int) using parquet")
spark.sql("create table t2(x int, y int, z int) using parquet")
spark.sql("select a from t1 left join t2 on a = x and b = y limit 5").explain("cost")
{code}

Current:
{noformat}
== Optimized Logical Plan ==
GlobalLimit 5
+- LocalLimit 5
   +- Project [a#0]
      +- Join LeftOuter, ((a#0 = x#3) AND (b#1 = y#4))
         :- Project [a#0, b#1]
         :  +- Relation default.t1[a#0,b#1,c#2] parquet
         +- Project [x#3, y#4]
            +- Filter (isnotnull(x#3) AND isnotnull(y#4))
               +- Relation default.t2[x#3,y#4,z#5] parquet
{noformat}

Excepted:
{noformat}
== Optimized Logical Plan ==
GlobalLimit 5
+- LocalLimit 5
   +- Project [a#0]
      +- Join LeftOuter, ((a#0 = x#3) AND (b#1 = y#4))
         :- LocalLimit 5
         :  +- Project [a#0, b#1]
         :     +- Relation default.t1[a#0,b#1,c#2] parquet
         +- Project [x#3, y#4]
            +- Filter (isnotnull(x#3) AND isnotnull(y#4))
               +- Relation default.t2[x#3,y#4,z#5] parquet
{noformat}
Bug: ProcfsMetricsGetter.computeAllMetrics may return partial metrics when some of child pids metrics are missing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/21 21:43
Affects Version/s: 3.0.0

When the procfs metrics of some child pids are unavailable, ProcfsMetricsGetter.computeAllMetrics() may return partial metrics (the sum of a subset of child pids), instead of an all 0 result. This can be misleading and is undesired per the current code comments in [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala#L214].

How to reproduce it?

This unit test is kind of self-explanatory:
{code:java}
    val p = new ProcfsMetricsGetter(getTestResourcePath("ProcfsMetrics"))
    val mockedP = spy(p)

    // proc file of pid 22764 doesn't exist, so partial metrics shouldn't be returned
    var ptree = Set(26109, 22764, 22763)
    when(mockedP.computeProcessTree).thenReturn(ptree)
    var r = mockedP.computeAllMetrics
    assert(r.jvmVmemTotal == 0)
    assert(r.jvmRSSTotal == 0)
    assert(r.pythonVmemTotal == 0)
    assert(r.pythonRSSTotal == 0)
{code}
In the current implementation, computeAllMetrics will reset the allMetrics to 0 when processing 22764 because 22764's proc file doesn't exist, but then it will continue processing pid 22763, and update allMetrics to procfs metrics of pid 22763.

Also, a side effect of this bug is that it can lead to a verbose warning log if many pids' stat files are missing. An early terminating can make the warning logs more concise.

How to solve it?

The issue can be fixed by throwing IOException to computeAllMetrics(), in that case, computeAllMetrics can aware that at lease one child pid's procfs metrics is missing and then terminate the metrics reporting.
Bug: Improve error message for casting cause overflow error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/21 14:50
Affects Version/s: 3.2.0

For example:
{code:sql}
set spark.sql.ansi.enabled=true;
select tinyint(128) * tinyint(2);
{code}

Error message:
{noformat}
Casting 128 to scala.Byte$ causes overflow
{noformat}

Expected:
{noformat}
Casting 128 to tinyint causes overflow
{noformat}

We should use DataType's catalogString.
Bug: Avoid wrapped in withNewExecutionId twice when run SQL with side effects
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Mar/21 06:13
Affects Version/s: 3.2.0

In the current code, a `QueryExecution` will be wrapped in `SQLExecution.withNewExecutionId` twice when run SQL with side effects.

To reproduce:

Run SQLs below in spark-sql
{code:java}
spark-sql> CREATE TABLE t AS SELECT 1 AS i;
spark-sql> SELECT * FROM t;
spark-sql> DROP TABLE t;

{code}
we will see duplicate SQL queries in Spark UI

!0F9E09B0-7FAF-4424-9C11-66930F1BBB7B.png!
Bug: Support for Parquet unsigned LogicalTypes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/21 03:19
Affects Version/s: 3.2.0

Parquet supports some unsigned datatypes.  Here is the definition related in parquet.thrift

{code:java}
/**
 * Common types used by frameworks(e.g. hive, pig) using parquet.  This helps map
 * between types in those frameworks to the base types in parquet.  This is only
 * metadata and not needed to read or write the data.
 */

  /**
   * An unsigned integer value.
   *
   * The number describes the maximum number of meaningful data bits in
   * the stored value. 8, 16 and 32 bit values are stored using the
   * INT32 physical type.  64 bit values are stored using the INT64
   * physical type.
   *
   */
  UINT_8 = 11;
  UINT_16 = 12;
  UINT_32 = 13;
  UINT_64 = 14;
{code}

Spark does not support unsigned datatypes. In SPARK-10113, we emit an exception with a clear message for them. 

UInt8-[0:255]
UInt16-[0:65535]
UInt32-[0:4294967295]
UInt64-[0:18446744073709551615]

Unsigned types - may be used to produce smaller in-memory representations of the data. If the stored value is larger than the maximum allowed by int32 or int64, then the behavior is undefined.

In this ticket, we try to read them as a higher precision signed type
Bug: AnsiTypeCoercion: return narrowest convertible type among TypeCollection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/21 04:52
Affects Version/s: 3.2.0

Currently, when implicit casting a data type to a `TypeCollection`, Spark returns the first convertible data type among `TypeCollection`.
In ANSI mode, we can make the behavior more reasonable by returning the narrowest convertible data type in `TypeCollection`.

In details,  we first try to find the all the expected types we can implicitly cast:
1. if there is no convertible data types, return None;
2. if there is only one convertible data type, cast input as it;
3. otherwise if there are multiple convertible data types, find the narrowest common data
     type among them. If there is no such narrowest common data type, return None.

Note that if the narrowest common type is Float type and the convertible types contains Double ype, simply return Double type as the narrowest common type to avoid potential
 precision loss on converting the Integral type as Float type.
Bug: Support TimestampType in UDT
Status: Open | Priority: Major | Resolution: 
Created: 23/Mar/21 13:50
Affects Version/s: 3.0.2

For user defined type with TimestampType, conversion from pandas to pyarrow and from pyarrow to Spark UnsafeRow needs to be carefully handled.


*From pandas to pyarrow*

{code:java}
if isinstance(dt, UserDefinedType):
    s = s.apply(dt.serialize)

t = to_arrow_type(dt)

array = pa.StructArray.from_pandas(s, mask=mask, type=t, safe=self._safecheck)
{code}

The above code may not work. If UDT is like:
{code:java}
class ExamplePointWithTimeUDT(UserDefinedType):
    """
    User-defined type (UDT) for ExamplePointWithTime.
    """

    @classmethod
    def sqlType(self):
        return StructType([
            StructField("x", DoubleType(), False),
            StructField("y", DoubleType(), True),
            StructField("ts", TimestampType(), False),
        ])

    @classmethod
    def module(cls):
        return 'pyspark.sql.tests'

    @classmethod
    def scalaUDT(cls):
        return 'org.apache.spark.sql.test.ExamplePointWithTimeUDT'

    def serialize(self, obj):
        return [obj.x, obj.y, obj.ts]

    def deserialize(self, datum):
        return ExamplePointWithTime(datum[0], datum[1], datum[2])


class ExamplePointWithTime:
    """
    An example class to demonstrate UDT in Scala, Java, and Python.
    """

    __UDT__ = ExamplePointWithTimeUDT()

    def __init__(self, x, y, ts):
        self.x = x
        self.y = y
        self.ts = ts

    def __repr__(self):
        return "ExamplePointWithTime(%s,%s,%s)" % (self.x, self.y, self.ts)

    def __str__(self):
        return "(%s,%s,%s)" % (self.x, self.y, self.ts)

    def __eq__(self, other):
        return isinstance(other, self.__class__) \
            and other.x == self.x and other.y == self.y \
            and other.ts == self.ts
{code}


*From pyarrow to Spark Internal*
Serialize and deserialize will fail.

See the failed PR demo: https://github.com/eddyxu/spark/pull/4
Bug: FileNotFoundException on _temporary when multiple app write to same table
Status: Open | Priority: Major | Resolution: 
Created: 23/Mar/21 15:24
Affects Version/s: 2.4.0

When multiple Spark applications are writing to the same hive table ( but different partitions so not interfering with each other in any way), the application finishing first ends up deleting the parent _temporary directory which is still being used by other application.

I think the temporary directory being used by FileOutputCommitter should be made configurable to let the caller call with with its own unique value as per the requirement and without having to worry about some other application deleting it unknowingly. Something like:
{quote}
 public static final String PENDING_DIR_NAME_DEFAULT = "_temporary";
 public static final String PENDING_DIR_NAME_DEFAULT =
 "mapreduce.fileoutputcommitter.tempdir";
{quote}
 

We can not use mapreduce.fileoutputcommitter.algorithm.version = 2 due to its issue with data loss https://issues.apache.org/jira/browse/MAPREDUCE-7282. 

There are similar Jira https://issues.apache.org/jira/browse/SPARK-18883, whihc was not resolved. This is very generic case of simply one spark application messing up working of other spark application working on same table and can be avoided by making temp unique or configurable.
Bug: K8s Integration test failed (due to libldap installation failed)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Mar/21 07:34
Affects Version/s: 3.1.0

Err:20

[http://security.debian.org/debian-security]

buster/updates/main amd64 libldap-common all 2.4.47+dfsg-3+deb10u4 404 Not Found [IP: 151.101.194.132 80] Err:21

[http://security.debian.org/debian-security]

buster/updates/main amd64 libldap-2.4-2 amd64 2.4.47+dfsg-3+deb10u4 404 Not Found [IP: 151.101.194.132 80] E: Failed to fetch

[http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-common_2.4.47+dfsg-3+deb10u4_all.deb]

404 Not Found [IP: 151.101.194.132 80] E: Failed to fetch

[http://security.debian.org/debian-security/pool/updates/main/o/openldap/libldap-2.4-2_2.4.47+dfsg-3+deb10u4_amd64.deb]

404 Not Found [IP: 151.101.194.132 80]

[1] http://mail-archives.apache.org/mod_mbox/spark-dev/202103.mbox/%3CCAGFcPdZY_TZ-qD7_SLvN5%2B1jjt9cp4GyTwxbXHbVHnD-stLSqw%40mail.gmail.com%3E
Bug: Fix collectToPython timeouts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/21 09:38
Affects Version/s: 2.4.7

One of our customers frequently encounters "serve-DataFrame" java.net.SocketTimeoutException: Accept timed errors in PySpark because DataSet.collectToPython() in Spark 2.4 does the following:


#     Collects the results
#     Opens up a socket server that is then listening to the connection from Python side
#     Runs the event listeners as part of withAction on the same thread as SPARK-25680 is not available in Spark 2.4
#     Returns the address of the socket server to Python
#     The Python side connects to the socket server and fetches the data

As the customer has a custom, long running event listener the time between 2. and 5. is frequently longer than the default connection timeout and increasing the connect timeout is not a good solution as we don't know how long running the listeners can take.
Bug: hive-common should exclusion hadoop related jars
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 22/Mar/21 08:50
Affects Version/s: 3.2.0

hive-common should exclusion hadoop related jars
{code:java}
      <scope>${hive.common.scope}</scope>
      <exclusions>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-auth</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-annotations</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-common</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
{code}