av9ash/GitBugs huggingface dataset:
Bug: Support cleaning up shuffle blocks from external shuffle service
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/21 15:40
Affects Version/s: 3.2.0

Currently shuffle data is not cleaned up when an external shuffle service is used and the associated executor has been deallocated before the shuffle is cleaned up. Shuffle data is only cleaned up once the application ends.

There have been various issues filed for this:

https://issues.apache.org/jira/browse/SPARK-26020

https://issues.apache.org/jira/browse/SPARK-17233

https://issues.apache.org/jira/browse/SPARK-4236

But shuffle files will still stick around until an application completes. Dynamic allocation is commonly used for long running jobs (such as structured streaming), so any long running jobs with a large shuffle involved will eventually fill up local disk space. The shuffle service already supports cleaning up shuffle service persisted RDDs, so it should be able to support cleaning up shuffle blocks as well once the shuffle is removed by the ContextCleaner. 

The current alternative is to use shuffle tracking instead of an external shuffle service, but this is less optimal from a resource perspective as all executors must be kept alive until the shuffle has been fully consumed and cleaned up (and with the default GC interval being 30 minutes this can waste a lot of time with executors held onto but not doing anything).
Bug: decouple amplab jenkins from spark website, builds and tests
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/21 01:15
Affects Version/s: 3.3.0

we will be turning off jenkins on dec 23rd, and we need to decouple the build infra from jenkins, as well as remove any amplab jenkins-specific docs on the website, scripts and infra setup.

i'll be creating > 1 PRs for this.
Bug: inject an early scan pushdown rule
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Dec/21 10:24
Affects Version/s: 3.3.0

Currently, Spark supports push down filters, aggregates and limit. All the job is completed by V2ScanRelationPushDown.
But, V2ScanRelationPushDown have a lot limit.
Users want apply custom rule for push down after V2ScanRelationPushDown failed.
Bug: Spark-3.2.0 Fetch Hbase Data not working
Status: Open | Priority: Major | Resolution: 
Created: 16/Dec/21 07:49
Affects Version/s: 3.2.0

Below is the sample code snipet that is used to fetch data from hbase. This used to work fine with spark-3.1.1

However after upgrading to psark-3.2.0 it is not working, The issue is it is not throwing any exception, it just don't fill RDD.

 
{code:java}
 
   def getInfo(sc: SparkContext, startDate:String, cachingValue: Int, sparkLoggerParams: SparkLoggerParams, zkIP: String, zkPort: String): RDD[(String)] = {{
val scan = new Scan
    scan.addFamily("family")
    scan.addColumn("family","time")
    val rdd = getHbaseConfiguredRDDFromScan(sc, zkIP, zkPort, "myTable", scan, cachingValue, sparkLoggerParams)
    val output: RDD[(String)] = rdd.map { row =>
      (Bytes.toString(row._2.getRow))
    }
    output
  }
 
def getHbaseConfiguredRDDFromScan(sc: SparkContext, zkIP: String, zkPort: String, tableName: String,
                                    scan: Scan, cachingValue: Int, sparkLoggerParams: SparkLoggerParams): NewHadoopRDD[ImmutableBytesWritable, Result] = {
    scan.setCaching(cachingValue)
    val scanString = Base64.getEncoder.encodeToString(org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(scan).toByteArray)
    val hbaseContext = new SparkHBaseContext(zkIP, zkPort)
    val hbaseConfig = hbaseContext.getConfiguration()
    hbaseConfig.set(TableInputFormat.INPUT_TABLE, tableName)
    hbaseConfig.set(TableInputFormat.SCAN, scanString)
    sc.newAPIHadoopRDD(
      hbaseConfig,
      classOf[TableInputFormat],
      classOf[ImmutableBytesWritable], classOf[Result]
    ).asInstanceOf[NewHadoopRDD[ImmutableBytesWritable, Result]]
  }
 
{code}
 

If we fetch with using scan directly without using newAPIHadoopRDD, it works.
Bug: Recursive view `df` detected (cycle: `df` -> `df`)
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/Dec/21 07:49
Affects Version/s: 3.2.0

In Spark 3.2.0, you can no longer reuse the same name for a temporary view.  This change is backwards incompatible, and means a common way of running pipelines of SQL queries no longer works.   The following is a simple reproducible example that works in Spark 2.x and 3.1.2, but not in 3.2.0: 

{code:python}from pyspark.context import SparkContext 
from pyspark.sql import SparkSession 
sc = SparkContext.getOrCreate() 
spark = SparkSession(sc) 
sql = """ SELECT id as col_1, rand() AS col_2 FROM RANGE(10); """ 

df = spark.sql(sql) 

df.createOrReplaceTempView("df") 

sql = """ SELECT * FROM df """ 

df = spark.sql(sql) 

df.createOrReplaceTempView("df") 

sql = """ SELECT * FROM df """ 

df = spark.sql(sql) {code}   

The following error is now produced:   
{code:python}AnalysisException: Recursive view `df` detected (cycle: `df` -> `df`) 
{code} 

I'm reasonably sure this change is unintentional in 3.2.0 since it breaks a lot of legacy code, and the `createOrReplaceTempView` method is named explicitly such that replacing an existing view should be allowed.   An internet search suggests other users have run into a similar problems, e.g. [here|https://community.databricks.com/s/question/0D53f00001Qugr7CAB/upgrading-from-spark-24-to-32-recursive-view-errors-when-using]
Bug: spark on k8s, when the user want to push python3.6.6.zip to the pod , but no permission to execute
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/21 12:17
Affects Version/s: 3.2.0

In cluster mode, I hava another question that when I unzip python3.6.6.zip in pod , but no permission to execute, my execute operation as follows：


{code:sh}
spark-submit \
--archives ./python3.6.6.zip#python3.6.6 \
--conf "spark.pyspark.python=python3.6.6/python3.6.6/bin/python3" \
--conf "spark.pyspark.driver.python=python3.6.6/python3.6.6/bin/python3" \
--conf spark.kubernetes.container.image.pullPolicy=Always \
./examples/src/main/python/pi.py 100
{code}
Bug: sql hang at planning stage
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/21 11:11
Affects Version/s: 3.1.1

when exec a sql, this sql hang at planning stage.

when disable DPP, sql can finish very quickly.

we can reproduce this  problem through example below:

create table test.test_a (
day string,
week int,
weekday int)
partitioned by (
dt varchar(8))
stored as orc;

insert into test.test_a partition (dt=20211126) values('1',1,2);

create table test.test_b (
session_id string,
device_id string,
brand string,
model string,
wx_version string,
os string,
net_work_type string,
app_id string,
app_name string,
col_z string,
page_url string,
page_title string,
olabel string,
otitle string,
source string,
send_dt string,
recv_dt string,
request_time string,
write_time string,
client_ip string,
col_a string,
dt_hour varchar(12),
product string,
channelfrom string,
customer_um string,
kb_code string,
col_b string,
rectype string,
errcode string,
col_c string,
pageid_merge string)
partitioned by (
dt varchar(8))
stored as orc;

insert into test.test_b partition(dt=20211126)
values('2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2','2');

 

 

drop table if exists test.test_c;create table if not exists test.test_c stored as ORCFILE as
select calendar.day,calendar.week,calendar.weekday, a_kbs,
b_kbs, c_kbs,d_kbs,e_kbs,f_kbs,g_kbs,h_kbs,i_kbs,j_kbs,k_kbs
from (select * from test.test_a where dt = '20211126') calendar
left join
(select dt,count(distinct kb_code) as a_kbs
from test.test_b
where dt = '20211126'
group by dt) t1
on calendar.dt = t1.dt

left join
(select dt,count(distinct kb_code) as b_kbs
from test.test_b
where dt = '20211126'
group by dt) t2
on calendar.dt = t2.dt

left join
(select dt,count(distinct kb_code) as c_kbs
from test.test_b
where dt = '20211126'
group by dt) t3
on calendar.dt = t3.dt

left join
(select dt,count(distinct kb_code) as d_kbs
from test.test_b
where dt = '20211126'
group by dt) t4
on calendar.dt = t4.dt

left join
(select dt,count(distinct kb_code) as e_kbs
from test.test_b
where dt = '20211126'
group by dt) t5
on calendar.dt = t5.dt

left join
(select dt,count(distinct kb_code) as f_kbs
from test.test_b
where dt = '20211126'
group by dt) t6
on calendar.dt = t6.dt

left join
(select dt,count(distinct kb_code) as g_kbs
from test.test_b
where dt = '20211126'
group by dt) t7
on calendar.dt = t7.dt

left join
(select dt,count(distinct kb_code) as h_kbs
from test.test_b
where dt = '20211126'
group by dt) t8
on calendar.dt = t8.dt

left join
(select dt,count(distinct kb_code) as i_kbs
from test.test_b
where dt = '20211126'
group by dt) t9
on calendar.dt = t9.dt

left join
(select dt,count(distinct kb_code) as j_kbs
from test.test_b
where dt = '20211126'
group by dt) t10
on calendar.dt = t10.dt

left join
(select dt,count(distinct kb_code) as k_kbs
from test.test_b
where dt = '20211126'
group by dt) t11
on calendar.dt = t11.dt
Bug: insert overwrite table but the partition information stored in Metastore was not changed
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 02/Dec/21 13:54
Affects Version/s: 3.2.0

I create a partitioned table in SparkSQL, insert a data entry, add a regular field, and finally insert a new data entry into the partition,The query is normal in SparkSQL, but the return value of the newly inserted field is NULL in Hive 2.3.9

for example

create table updata_col_test1(a int) partitioned by (dt string); 
insert overwrite table updata_col_test1 partition(dt='20200101') values(1); 
insert overwrite table updata_col_test1 partition(dt='20200102') values(1);
insert overwrite table updata_col_test1 partition(dt='20200103') values(1);

alter table  updata_col_test1 add columns (b int);

insert overwrite table updata_col_test1 partition(dt) values(1, 2, '20200101'); fail
insert overwrite table updata_col_test1 partition(dt='20200101') values(1, 2); fail
insert overwrite table updata_col_test1 partition(dt='20200104') values(1, 2); sucessfully
Bug: SparkSession.setLogLevel not working in Spark Shell
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/21 12:10
Affects Version/s: 3.3.0

In Spark 3.2:

{code}
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@7dafb9f9

scala> spark.sparkContext.setLogLevel("WARN")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
21/12/23 21:08:18 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@7dafb9f9

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res5: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@7dafb9f9
{code}

In the current master:

{code}
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@3e8a1137

scala> spark.sparkContext.setLogLevel("WARN")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@3e8a1137

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res5: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@3e8a1137
{code}

Seems like it works when you set via {{setLogLevel}} initially but cannot be changed afterward.
Bug: Show ignored confs & hide warnings for conf already set in SparkSession.builder.getOrCreate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/21 11:41
Affects Version/s: 3.3.0

Currently, {{SparkSession.builder.getOrCreate()}} is too noisy even when duplicate configurations are set. And users cannot tell which configurations are to fix. See the example below:

{code}
./bin/spark-shell --conf spark.abc=abc
{code}

{code}
import org.apache.spark.sql.SparkSession
spark.sparkContext.setLogLevel("DEBUG")
SparkSession.builder.config("spark.abc", "abc").getOrCreate
{code}

{code}
...
21:12:40.601 [main] WARN  org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
{code}

This is strait forward when there are few configurations but it is difficult for users to figure out when there are too many configurations especially when these configurations are defined in property files like {{spark-default.conf}} that is sometimes maintained separately by system admins.

See also https://github.com/apache/spark/pull/34757#discussion_r769248275
Bug: Support str and timestamp for (Series|DataFrame).describe()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/21 02:09
Affects Version/s: 3.3.0

Initialized in Koalas issue: [https://github.com/databricks/koalas/issues/1888]

 

The `(Series|DataFrame).describe()` in pandas API on Spark doesn't work properly when DataFrame has no numeric column.

 

 
{code:java}
>>> df = ps.DataFrame({'a': ["a", "b", "c"]})
>>> df.describe()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../python/pyspark/pandas/frame.py", line 7582, in describe
    raise ValueError("Cannot describe a DataFrame without columns")
ValueError: Cannot describe a DataFrame without columns 
{code}
 

As it works fine in pandas, we should fix it.
Bug: pyspark adding third-party Dependencies on k8s
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 21/Dec/21 17:54
Affects Version/s: 3.2.0

I have a question about that how do I add my Python dependencies to Spark Job, as following


{code:sh}
spark-submit \
--archives s3a://path/python3.6.9.tgz#python3.6.9 \
--conf "spark.pyspark.driver.python=python3.6.9/bin/python3" \
--conf "spark.pyspark.python=python3.6.9/bin/python3" \
--name "piroottest" \
./examples/src/main/python/pi.py 10
{code}

this can't run my job sucessfully，it throw error


{code:sh}
Traceback (most recent call last):
  File "/tmp/spark-63b77184-6e89-4121-bc32-6a1b793e0c85/pi.py", line 21, in <module>
    from pyspark.sql import SparkSession
  File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 121, in <module>
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/__init__.py", line 42, in <module>
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 27, in <module>
    async def _ag():
  File "/opt/spark/work-dir/python3.6.9/lib/python3.6/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.6: cannot open shared object file: No such file or directory
{code}

Or is there another way to add Python dependencies？
Bug: 'Index' object has no attribute 'levels' in  pyspark.pandas.frame.DataFrame.insert
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/21 20:49
Affects Version/s: 3.3.0

 [This piece of code|https://github.com/apache/spark/blob/6e45b04db48008fa033b09df983d3bd1c4f790ea/python/pyspark/pandas/frame.py#L3991-L3993] in {{pyspark.pandas.frame}} is going to fail on runtime, when {{is_name_like_tuple}} evaluates to {{True}}

{code:python}
if is_name_like_tuple(column):
    if len(column) != len(self.columns.levels):
{code}

with 

{code}
'Index' object has no attribute 'levels'
{code}

To be honest, I am not sure what is intended behavior (initially, I suspected that we should have 

{code:python}
 if len(column) != self.columns.nlevels
{code}

but {{nlevels}} is hard-coded to one, and wouldn't be consistent with Pandas at all.
Bug: Failed to execute pyspark test in Win WSL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/21 04:10
Affects Version/s: 3.3.0

 
{code:java}
Launching unittests with arguments python -m unittest test_rdd.RDDTests.test_range in /home/yikun/spark/python/pyspark/testsTraceback (most recent call last):
  File "/mnt/d/Program Files/JetBrains/PyCharm 2021.1.3/plugins/python/helpers/pycharm/_jb_unittest_runner.py", line 35, in <module>
    sys.exit(main(argv=args, module=None, testRunner=unittestpy.TeamcityTestRunner, buffer=not JB_DISABLE_BUFFERING))
  File "/usr/lib/python3.8/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.8/unittest/main.py", line 147, in parseArgs
    self.createTests()
  File "/usr/lib/python3.8/unittest/main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "/usr/lib/python3.8/unittest/loader.py", line 220, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.8/unittest/loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python3.8/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/home/yikun/spark/python/pyspark/tests/test_rdd.py", line 37, in <module>
    from pyspark.testing.utils import ReusedPySparkTestCase, SPARK_HOME, QuietTest
  File "/home/yikun/spark/python/pyspark/testing/utils.py", line 47, in <module>
    SPARK_HOME = os.environ["SPARK_HOME"]#_find_spark_home()
  File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'SPARK_HOME' {code}
Looks like we  should change "SPARK_HOME = os.environ["SPARK_HOME"]" to "SPARK_HOME = _find_spark_home()"
Bug: Deserializing problem in java 17
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/21 13:04
Affects Version/s: 3.2.0

When using the connector with Spark , there is a problem with deserializing using databricks jdbc driver in java 17 ( https://databricks.com/spark/jdbc-drivers-download)

Caused by: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available

 

Setting parameter "-Dio.netty.tryReflectionSetAccessible=true" doesn't help.
Bug: rolled event log still need be clean after compact
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/21 06:56
Affects Version/s: 3.1.1

when we set "{{{}spark.eventLog.rolling.enabled{}}} =true", the eventlog will be roll and compact(when set "spark.eventLog.compression.codec"), the directory tree like this

root dir: /spark2xJobHistory2x/eventlog_v2_application_xxxxxxxxxxx_xxx_1

file in dir:

 /spark2xJobHistory2x/eventlog_v2_application_xxxx_xxx_1/events_xxxx_application_xxxxxxxxxxx_xxxx_1.zstd

 /spark2xJobHistory2x/eventlog_v2_application_xxxx_xxx_1/events_xxxx_application_xxxxxxxxxxx_xxxx_1.zstd

 /spark2xJobHistory2x/eventlog_v2_application_xxxx_xxx_1/events_xxxx_application_xxxxxxxxxxx_xxxx_1.zstd

......

......

 

a "long run" spark application, the history server will not clean the 'events_xxxx_application_xxxxxxxxxxx_xxxx_1.zstd' file in /spark2xJobHistory2x/eventlog_v2_application_xxxxxxxxxxx_xxx_1, so the size of directory will be bigger and bigger during the whole lifetime of app. 

so i think we should provide a mechanism for user to clean the “events_xxxx_application_xxxxxxxxxxx_xxxx_1.zstd” file in /spark2xJobHistory2x/eventlog_v2_application_xxxxxxxxxxx_xxx_1 directory

 

our solution：add a clean function in “[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#checkForLogs]”，this function will list the file in “/spark2xJobHistory2x/eventlog_v2_application_xxxxxxxxxxx_xxx_1” and clean the “events_xxxx_application_xxxxxxxxxxx_xxxx_1.zstd” file according to the config "spark.history.fs.cleaner.maxAge". this solve the unlimited space increase ，but will loss some event，especially the start event，this will lead the history can not show the eventlog correctly。

 

so we will will have a much more proper way to solve this
Bug: Upgrade SBT to 1.5.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/21 01:53
Affects Version/s: 3.2.1

SBT 1.5.7 was released a few hours ago, which includes a fix for CVE-2021-45046.
https://github.com/sbt/sbt/releases/tag/v1.5.7
Bug: Avoid touching Scala reflection APIs in the lit function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/21 18:05
Affects Version/s: 3.2.0

Currently lit is slow when the concurrency is high as it needs to hit the Scala reflection code which hits global locks. For example, running the following test locally using Spark 3.2 shows the difference:
{code:java}
scala> :paste
// Entering paste mode (ctrl-D to finish)import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
import org.apache.spark.sql.catalyst.expressions.Literalval parallelism = 50def testLiteral(): Unit = {
  val ts = for (_ <- 0 until parallelism) yield {
    new Thread() {
      override def run() {
         for (_ <- 0 until 50) {
          new Column(Literal(0L))
        }
      }
    }
  }
  ts.foreach(_.start())
  ts.foreach(_.join())
}def testLit(): Unit = {
  val ts = for (_ <- 0 until parallelism) yield {
    new Thread() {
      override def run() {
         for (_ <- 0 until 50) {
          lit(0L)
        }
      }
    }
  }
  ts.foreach(_.start())
  ts.foreach(_.join())
}println("warmup")
testLiteral()
testLit()println("lit: false")
spark.time {
  testLiteral()
}
println("lit: true")
spark.time {
  testLit()
}// Exiting paste mode, now interpreting.warmup
lit: false
Time taken: 8 ms
lit: true
Time taken: 682 ms
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
import org.apache.spark.sql.catalyst.expressions.Literal
parallelism: Int = 50
testLiteral: ()Unit
testLit: ()Unit {code}