av9ash/GitBugs huggingface dataset:
Bug: SparkR not on CRAN (again)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 18:41
Affects Version/s: 2.4.7

SparkR is removed from CRAN 

[https://cran.r-project.org/web/packages/SparkR/index.html#:~:text=Package%20'SparkR'%20was%20removed%20from,be%20obtained%20from%20the%20archive.&text=A%20summary%20of%20the%20most,to%20link%20to%20this%20page.]

Is there anything we can do to help?
Bug: Keep behavior consistent when conf `spark.sql.legacy.parser.havingWithoutGroupByAsWhere` is true with migration guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 12:11
Affects Version/s: 2.4.8

According to [https://github.com/apache/spark/pull/29087#issuecomment-754389257]

fix bug and add UT
Bug: NOW is interpreted as the NOW SQL function
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 29/Jan/21 15:56
Affects Version/s: 3.0.0

I think we ran into a bug in the Spark framework. Basically, the bug we caught is like this: when reading a data frame in Parquet format partitioned by a column, if the column contains values of “NOW”, NOW will be interpreted as the NOW function as in SQL, and returns the literal timestamp of NOW.

 

Steps to reproduce:

from pyspark.sql.session import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame([['NOW', 1], ['THEN', 2]], schema=['Col1', 'Col2'])

df.write.parquet('/tmp/my_partitioned_data', mode='overwrite', partitionBy=['Col1'])

df_read_back = spark.read.parquet('/tmp/my_partitioned_data')

"""
In [1]: df.show()
+----+----+
|Col1|Col2|
+----+----+
| NOW| 1|
|THEN| 2|
+----+----+


In [2]: df_read_back.show()
+----+--------------------+
|Col2| Col1|
+----+--------------------+
| 1|2021-01-22 10:46:...|
| 2| THEN|
+----+--------------------+
Bug: Implement Parquet StringEndsWith、StringContains Filter
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 29/Jan/21 04:57
Affects Version/s: 3.0.0

When create parquetFilters, currently only implements  

{code:java}
case sources.StringStartsWith(name, prefix)
{code}

But there exists StringEndsWith、StringContains in /spark/sql/catalyst/src/main/scala/org/apache/spark/sql/sources/filters.scala

We can implements this two filters, and  rename 

{code:java}
PARQUET_FILTER_PUSHDOWN_STRING_STARTSWITH_ENABLED 
{code}

 to


{code:java}
PARQUET_FILTER_PUSHDOWN_STRING_ENABLED 
{code}
Bug: Document CREATE EXTERNAL datasource TABLE
Status: Open | Priority: Major | Resolution: 
Created: 27/Jan/21 05:47
Affects Version/s: 3.1.0

We should mention what EXTERNAL means in CREAT TABLE data source one as well at https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-ddl-create-table-datasource.md.
It should also include SPARK-33651
Bug: Unexpected outcome from Python UDF where its return type is MapType
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 28/Jan/21 04:40
Affects Version/s: 3.0.1

I tried to create a Python UDF for sorting a column with `MapType` column by its key. Yet the final result seems not sorted as expected.
{code:python|title=Initialization}
from pyspark.sql.types import MapType, StructType, StructField, StringType, DoubleType, IntegerType
from pyspark.sql import Row

schema = StructType([
    StructField("Name", StringType(), True),
    StructField("Map_Value", MapType(IntegerType(), DoubleType(), False), False)])
test_data = [
    Row(name='sample_1', LD_Value={10: 0.20, 5: 0.30, 17: 0.25}),
    Row(name='sample_2', LD_Value={11: 0.40, 6: 0.67, 24: 0.45})
]
base = spark.createDataFrame(test_data, schema)

# define a Python UDF for sorting map-type columns
def sort_maptype_function(col, reverse=False):
    sorted_col = dict(sorted(col.items(), key=lambda item: item[0], reverse=reverse))
    print(sorted_col)
    return sorted_col
sort_map = spark.udf.register(
        "SORT_MAPTYPE_FUNCTION", 
        sort_maptype_function, 
        MapType(IntegerType(), DoubleType()))
result = base.selectExpr("Name", "Map_Value", "SORT_MAPTYPE_FUNCTION(LD_Value) AS Sorted_Map_Value")
{code}
{code:python|title=Display Result}
>>> result.show(truncate=100)
{5: 0.3, 10: 0.2, 17: 0.25}
{6: 0.67, 11: 0.4, 24: 0.45}
+--------+----------------------------------+----------------------------------+
|    Name|                         Map_Value|                  Sorted_Map_Value|
+--------+----------------------------------+----------------------------------+
|sample_1| [17 -> 0.25, 10 -> 0.2, 5 -> 0.3]| [17 -> 0.25, 10 -> 0.2, 5 -> 0.3]|
|sample_2|[24 -> 0.45, 11 -> 0.4, 6 -> 0.67]|[24 -> 0.45, 11 -> 0.4, 6 -> 0.67]|
+--------+----------------------------------+----------------------------------+
{code}
{code:python|title=Extended Explanation}
== Parsed Logical Plan ==
'Project [unresolvedalias('Name, None), unresolvedalias('Map_Value, None), 'SORT_MAPTYPE_FUNCTION('Map_Value) AS Sorted_Map_Value#192]
+- LogicalRDD [Name#169, Map_Value#170], false

== Analyzed Logical Plan ==
Name: string, Map_Value: map<int,double>, Sorted_Map_Value: map<int,double>
Project [Name#169, Map_Value#170, SORT_MAPTYPE_FUNCTION(Map_Value#170) AS Sorted_Map_Value#192]
+- LogicalRDD [Name#169, Map_Value#170], false

== Optimized Logical Plan ==
Project [Name#169, Map_Value#170, pythonUDF0#225 AS Sorted_Map_Value#192]
+- BatchEvalPython [SORT_MAPTYPE_FUNCTION(Map_Value#170)], [pythonUDF0#225]
   +- LogicalRDD [Name#169, Map_Value#170], false

== Physical Plan ==
*(2) Project [Name#169, Map_Value#170, pythonUDF0#225 AS Sorted_Map_Value#192]
+- BatchEvalPython [SORT_MAPTYPE_FUNCTION(Map_Value#170)], [pythonUDF0#225]
   +- *(1) Scan ExistingRDD[Name#169,Map_Value#170]
{code}
 

 

Why the result after the execution of the UDF for sorting map-type columns by its key still the same as the original column based on PySpark?
Bug: Spark handle always return LOST status in standalone cluster mode with Spark launcher
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 04/Jan/21 10:59
Affects Version/s: 2.4.4

I can use it to submit spark app successfully in standalone client/yarn client/yarn cluster mode，and get correct app status, but when i submit spark app in standalone cluster mode, Spark handle always return LOST status(once) and app running stablely until FINISHED( handle wasn't get any state change infomation).  I noticed when I submited app from code, after a while, the SparkSubmit process was suddenly stopped. I checked sparkSubmit log(launcher redirect log) doesn't have any useful information.

this is my pseudo code,
{code:java}
SparkAppHandle handle = launcher.startApplication(new SparkAppHandle.Listener() {
    @Override
    public void stateChanged(SparkAppHandle handle) {
        stateChangedHandle(handle.getAppId(), jobId, code, execId, handle.getState(), driverInfo, request, infoLog, errorLog);
    }
    @Override
    public void infoChanged(SparkAppHandle handle) {
        stateChangedHandle(handle.getAppId(), jobId, code, execId, handle.getState(), driverInfo, request, infoLog, errorLog);
    }
});{code}
any idea ? thx
Bug: SparkR partition timing summary reports input time correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/21 18:05
Affects Version/s: 2.3.2

When sparkR is run at log level INFO, a summary of how the worker spent its time processing the partition is printed. There is a logic error where it is over-reporting the time inputting rows.

In detail: the variable inputElap in a wider context is used to mark the beginning of reading rows, but in the part changed here it was used as a local variable for measuring compute time. Thus, the error is not observable if there is only one group per partition, which is what you get in unit tests.

For our application, here's what a log entry looks like before these changes were applied:

{{20/10/09 04:08:58 WARN RRunner: Times: boot = 0.013 s, init = 0.005 s, broadcast = 0.000 s, read-input = 529.471 s, compute = 492.037 s, write-output = 0.020 s, total = 1021.546 s}}

this indicates that we're spending more time reading rows than operating on the rows.

After these changes, it looks like this:

{{20/12/15 06:43:29 WARN RRunner: Times: boot = 0.013 s, init = 0.010 s, broadcast = 0.000 s, read-input = 120.275 s, compute = 1680.161 s, write-output = 0.045 s, total = 1812.553 s}}
Bug: SparkUI: Storage page is empty even if cached
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/21 07:56
Affects Version/s: 2.3.3

scala> import org.apache.spark.storage.StorageLevel
import org.apache.spark.storage.StorageLevel

scala> val rdd = sc.parallelize(1 to 10, 1).persist(StorageLevel.MEMORY_ONLY_2)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:25

scala> rdd.count
res0: Long = 10

but SparkUI storage page is empty
Bug: Sparksql does not support when the inserted table is a read table
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 04/Jan/21 08:11
Affects Version/s: 3.0.1

When the inserted table is a read table, sparksql will throw an error - > org.apache.spark . sql.AnalysisException : Cannot overwrite a path that is also being read from.;
Bug: Combine distinct unions in more cases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/21 19:43
Affects Version/s: 3.2.0

In several TPCDS queries the CombineUnions rule does not manage to combine unions, because they have noop Projects between them.
The Projects will be removed by RemoveNoopOperators, but by then ReplaceDistinctWithAggregate has been applied and there are aggregates between the unions.
Bug: Improve the statistics estimation of the Tail
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/21 09:29
Affects Version/s: 3.2.0

{code:scala}
spark.sql("set spark.sql.cbo.enabled=true")
spark.range(100).selectExpr("id as a", "id as b", "id as c", "id as e").write.saveAsTable("t1")
println(Tail(Literal(5), spark.sql("SELECT * FROM t1").queryExecution.logical).queryExecution.explainString(org.apache.spark.sql.execution.CostMode))

{code}

Current:
{noformat}
== Optimized Logical Plan ==
Tail 5, Statistics(sizeInBytes=3.8 KiB)
+- Relation[a#24L,b#25L,c#26L,e#27L] parquet, Statistics(sizeInBytes=3.8 KiB)
{noformat}

Expected:
{noformat}
== Optimized Logical Plan ==
Tail 5, Statistics(sizeInBytes=200.0 B, rowCount=5)
+- Relation[a#24L,b#25L,c#26L,e#27L] parquet, Statistics(sizeInBytes=3.8 KiB)
{noformat}
Bug: `isCached` return `false` for cached Hive table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/21 19:15
Affects Version/s: 3.1.0

The same works in Spark 3.0 but fails in Spark 3.2.0-SNAPSHOT (and 3.1.0 probably):
*Spark 3.0:*
{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/
scala> sql("CREATE TABLE tbl (col int)")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.catalog.isCached("tbl")
res3: Boolean = false

scala> sql("CACHE TABLE tbl")
res4: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res5: Boolean = true
{code}

*Spark master:*
{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.0-SNAPSHOT
      /_/
scala> sql("CREATE TABLE tbl (col int)")
res1: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res2: Boolean = false

scala> sql("CACHE TABLE tbl")
res3: org.apache.spark.sql.DataFrame = []

scala> spark.catalog.isCached("tbl")
res4: Boolean = false
{code}
Bug: ALTER TABLE .. SET LOCATION doesn't refresh v1 table cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/21 10:40
Affects Version/s: 3.0.2

The example below portraits the issue:
1. Create a source table:
{code:sql}
spark-sql> CREATE TABLE src_tbl (c0 int, part int) USING hive PARTITIONED BY (part);
spark-sql> INSERT INTO src_tbl PARTITION (part=0) SELECT 0;
spark-sql> SHOW TABLE EXTENDED LIKE 'src_tbl' PARTITION (part=0);
default	src_tbl	false	Partition Values: [part=0]
Location: file:/Users/maximgekk/proj/refresh-cache-set-location/spark-warehouse/src_tbl/part=0
...
{code}
2. Load data from the source table to a cached destination table:
{code:sql}
spark-sql> CREATE TABLE dst_tbl (c0 int, part int) USING hive PARTITIONED BY (part);
spark-sql> ALTER TABLE dst_tbl ADD PARTITION (part=0);
spark-sql> INSERT INTO dst_tbl PARTITION (part=1) SELECT 1;
spark-sql> CACHE TABLE dst_tbl;
spark-sql> SELECT * FROM dst_tbl;
1	1
spark-sql> ALTER TABLE dst_tbl PARTITION (part=0) SET LOCATION '/Users/maximgekk/proj/refresh-cache-set-location/spark-warehouse/src_tbl/part=0';
spark-sql> SELECT * FROM dst_tbl;
1	1
{code}
The last query does not show recently loaded data from the source table.
Bug: The Signature for ConcatWs in Spark SQL Docs Is Inconsistent with the Actual Behavior
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/21 18:44
Affects Version/s: 3.2.0

In [https://spark.apache.org/docs/latest/api/sql/index.html#concat_ws], the signature of concat_ws is concat_ws(sep, [str | array(str)]+). However, it doesn't actually need any str or array(str) arguments. Only having the sep argument seems to work.

For example, `select concat_ws("a")` returns an empty string.
{code:java}
scala> sql("""select concat_ws("a")""").show
+------------+
|concat_ws(a)|
+------------+
|            |
+------------+
{code}
Bug: AvroSuite has test failure when run from IDE due to bad loading of resource file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/21 21:58
Affects Version/s: 3.2.0

Within {{AvroSuite}} the test {{Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled}} attempts to load a resource file like:
{code}
        val srcFile = new File("src/test/resources/episodes.avro")
        val destFile = new File(dir, "episodes.avro")
        FileUtils.copyFile(srcFile, destFile)
{code}
This works okay when tests are run via Maven/sbt , but really isn't the right way to access a resource, causing failures when run via IntelliJ, and actually {{episodes.avro}} has already been loaded properly as {{episodesAvro}}. Switch over to loading it properly:

{code}
        Files.copy(
          Paths.get(new URL(episodesAvro).toURI),
          Paths.get(dir.getCanonicalPath, "episodes.avro"))
{code}
Bug: Make spark.sql.hive as a private package
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/21 05:49
Affects Version/s: 3.1.1

Following the comment [https://github.com/apache/spark/pull/31271#discussion_r562598983,] we need to make spark.sql.hive as a private package and remove them from the API documentation.
Bug: Add partition columns for TPCDS tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/21 07:49
Affects Version/s: 3.2.0

We should add partition columns for our TPCDS related benchmark, The partition column should refer to [TPCDSTables|https://github.com/databricks/spark-sql-perf/blob/master/src/main/scala/com/databricks/spark/sql/perf/tpcds/TPCDSTables.scala].
Bug: Spark Structured Streaming - Kafka avro transformation on optional field Failed
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/21 23:21
Affects Version/s: 2.4.7

Hello All,
I have a spark structured streaming job to inject data from Kafka where message from Kafka is avro type.
Some of the fields are optional in the data. And I have to perform transformation if those optional fields are present in the data. 
So I tried to check whether the column exists by :


{color:#0747A6}def has_column(dataframe, col):
    """
    This function checks the existence of a given column in the given DataFrame

    :param dataframe: the dataframe
    :type dataframe: DataFrame
    :param col: the column name
    :type col: str
    :return: true if the column exists else false
    :rtype: boolean
    """
    try:
        dataframe[col]
        return True
    except AnalysisException:
        return False{color}


But it seems not working when its a streaming dataframe, but when the dataframe is normal dataframe, and when a column is not present the above check returns false, therefore I can ignore the transformation on the missing column. 

But on Streaming dataframe *has_column* always returns true and therefore the transformation get executed and cause exception. What is the right approach to check existence of column in a streaming dataframe before performing transformation?

Why streaming dataframe and normal dataframe differ in behavior? How to skip transformation on a column if it doesn't exists?
Bug: udf type hint should allow dectorator with named returnType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 14:08
Affects Version/s: 3.1.0

At the moment annotations allow following decorated patterns:
 
{code:python}
@udf
def f(x): ...

@udf("string")  # Or DataType instance
def f(x): ...

@udf(f="string")  # Awkward but technically valid
def f(x): ...
{code}

We should also support 

{code:python}
@udf(returnType="string")
def f(x): ...
{code}
Bug: Supplement the description for Python Package Management
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 09:10
Affects Version/s: 3.1.0

There is lack of explanation for "Using Virtualenv" chapter.

It says "It packs the current virtual environment to an archive file, and It self-contains both Python interpreter and the dependencies", but it's not working if there is no Python installed on the node.

Because the Python in the packed environment has a symbolic link that connects Python to the local one, so Python must exist in the same path on all nodes.
Bug: Cannot create a record reader because of a previous error when spark accesses the hive on HBase table 
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Jan/21 03:50
Affects Version/s: 3.0.1

It is normal for version 2.4.6 to use spark SQL to access hive on HBase table,Upgrade to spark3.0.1 with the following exception：

java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous logs lines from the task's full log for more details.
 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:252)
 at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:131)
 at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
 at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
 at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)

Caused by: java.lang.IllegalStateException: The input format instance has not been properly initialized. Ensure you call initializeTable either in your constructor or initialize method
 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getTable(TableInputFormatBase.java:585)
 at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:247)
 ... 59 more
java.io.IOException: Cannot create a record reader because of a previous error. Please look at the previous logs lines from the task's full log for more details.
Bug: Unify v1 and v2 ALTER TABLE .. SET SERDE tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jan/21 16:45
Affects Version/s: 3.2.0

Extract ALTER TABLE .. SET SERDE tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: pyspark.ml.stat.Summarizer should allow sparse vector results
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 19/Jan/21 15:52
Affects Version/s: 3.0.1

currently pyspark.ml.stat.Summarizer will return a dense vector, even if the input is sparse.

the Summarizer should either deduce the relevant type from the input, or add a parameter that forces sparse output

code to reproduce:

{{import pyspark}}
{{from pyspark.sql.functions import col}}
{{from pyspark.ml.stat import Summarizer}}
{{from pyspark.ml.linalg import SparseVector, DenseVector}}{{sc = pyspark.SparkContext.getOrCreate()}}
{{sql_context = pyspark.SQLContext(sc)}}{{df = sc.parallelize([ ( SparseVector(100, \{1: 1.0}),)]).toDF(['v'])}}
{{print(df.head())}}
{{print(df.select(Summarizer.mean(col('v'))).head())}}

ouput:

{{Row(v=SparseVector(100, \{1: 1.0})) }}
{{Row(mean(v)=DenseVector([0.0, 1.0,}}
{{0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]))}}
Bug: Add RocksDB StateStore implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/21 00:43
Affects Version/s: 3.2.0

Currently Spark SS only has one built-in StateStore implementation HDFSBackedStateStore. Actually it uses in-memory map to store state rows. As there are more and more streaming applications, some of them requires to use large state in stateful operations such as streaming aggregation and join.

Several other major streaming frameworks already use RocksDB for state management. So it is proven to be good choice for large state usage. But Spark SS still lacks of a built-in state store for the requirement.

We would like to explore the possibility to add RocksDB-based StateStore into Spark SS.
Bug: Allow implicit casting string literal to other data types under ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/21 16:08
Affects Version/s: 3.1.2


As the default type coercion rules allow converting String type to other primitive types, we can specially allow converting string literal to other primitive types in ANSI mode. So that users won't get many SQL execution failure on switching to ANSI mode.
This is learned from PostgreSQL.
Bug: Upgrade ORC to 1.6.7
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Jan/21 23:13
Affects Version/s: 3.2.0

Apache ORC 1.6.7 has the following fixes including ORC-711 Support CryptoExtension in create/decryptLocalKey.
 * [https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318320&version=12349470]
Bug: Add pipe API to Dataset
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 22/Jan/21 18:05
Affects Version/s: 3.2.0

Dataset doesn't have pipe API but RDD has it. Although for normal Dataset, user can convert a Dataset to RDD and call RDD.pipe, for streaming Dataset it is not possible. So that being said, this is actually a requirement from Structured Streaming, but we need to add pipe API to Dataset to enable it in Structured Streaming.
Bug: Improve performance for nth_value ignore nulls over offset window
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 08:09
Affects Version/s: 3.2.0

The current 
{code:java}
UnboundedOffsetWindowFunctionFrame
{code}
and
{code:java}
UnboundedPrecedingOffsetWindowFunctionFrame
{code}
 only support nth_value that respect nulls. So nth_value will execute 
{code:java}
updateExpressions
{code}
 multiple times.
Bug: Improve performance for last_value over unbounded window frame
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 27/Jan/21 07:06
Affects Version/s: 3.2.0

The current implement of `last_value` over unbounded window frame will execute `updateExpressions` multiple times.
In fact, `last_value` only execute `updateExpressions` once on the last row.