av9ash/GitBugs huggingface dataset:
Bug: Use `isEmpty` to improve performance in Pregel's superstep
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/21 02:39
Affects Version/s: 3.3.0

When I was running `Graphx.connectedComponents` with 20+ billion vertices and edges, I found that count is very slow.
{code:java}
object Pregel extends Logging {
  ...
  def apply[VD: ClassTag, ED: ClassTag, A: ClassTag] (...): Graph[VD, ED] = {
    ...
    // Maybe messages.isEmpty() is better than messages.count()
    var activeMessages = messages.count()

    // Loop
    var prevG: Graph[VD, ED] = null
    var i = 0
    while (activeMessages > 0 && i < maxIterations) {
      ...
      activeMessages = messages.count()
      ...
    }
    ...
    g
  } // end of apply

} // end of class Pregel
{code}
Maybe we only need an action operator here and active-messages are not empty, so we don’t need to use count, it’s better to use isEmpty. I verified it and it worked very well.
Bug: Enhance the docs and tests of try_add/try_divide
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/21 09:11
Affects Version/s: 3.2.0

The try_add function allows the following inputs:
* number, number
* date, number
* date, interval
* timestamp, interval
* interval, interval

And, the try_divide function allows the following inputs:
* number, number
* interval, number

However, in the current code, there are only examples and tests about the (number, number) inputs. We should enhance the docs to let users know that the functions can be used for datetime and interval operations too.
Bug: Upgrade commons-pool2 from 2.6.2 to 2.11.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Aug/21 09:07
Affects Version/s: 3.3.0

`2.11.1` is a Java 8 build version and `2.6.2` is still a Java 7 build version.

bug fix :
 * DefaultPooledObject.getIdleTime() drops nanoseconds on Java 9 and greater
 * POOL-361: Move validation for newly created objects into create(). Fixes #23
 * POOL-374: org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(K, T) should throw IllegalStateException instead of NullPointerException when a key is not found in the pool map.
 * POOL-376: Fixed regression from original fix for POOL-356 which could result in NPE when destroying objects.
 * POOL-326: Eliminated NPE / ISE exceptions due to keyed pools being prematurely removed.
 * Close BufferedOutputStream in test before calling toString on underlying BufferedOutputStream #26.
 * POOL-386: Refactored EvictionTimer usage tracking to fix POOL-386 and handle abandoned pools. #32
 * Make Duration setters use their respective default values when null
 * Call swallowException(Exception) instead of printing exceptions to the console in GenericKeyedObjectPool.removeAbandoned(AbandonedConfig) and GenericObjectPool.removeAbandoned(AbandonedConfig)
 * Getting a PooledObject's active duration returns a negative duration when the object is borrowed but not returned
Bug: Images used in pyspark documentation still use Koalas
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Aug/21 10:13
Affects Version/s: 3.2.0

Images in [Transform and apply a function|https://github.com/apache/spark/blob/master/python/docs/source/user_guide/pandas_on_spark/transform_apply.rst] documentation still uses the word Koalas, althogh the word was replaced to panas-on-Spark by this [PR|https://github.com/apache/spark/pull/32835] .

I think we have to match the word on that images
Bug: Deflake PySpark coverage report
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Aug/21 06:40
Affects Version/s: 3.2.0

https://github.com/apache/spark/runs/3388727798?check_suite_focus=true
https://github.com/apache/spark/runs/3392972609?check_suite_focus=true
https://github.com/apache/spark/runs/3359880048?check_suite_focus=true
https://github.com/apache/spark/runs/3338876122?check_suite_focus=true

PySpark scheduled coverage jobs are flaky. We should deflake them
Bug: Error message while trying to use spark sql functions directly on dataframe columns without using select expression
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Aug/21 02:29
Affects Version/s: 3.1.1

The below code generates a dataframe successfully . Here make_date function is used inside a select expression

 

from pyspark.sql.functions import  expr, make_date

df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])

df.select("*",expr("make_date(Y,M,D) as lk")).show()

 

The below code fails with a message "cannot import name 'make_date' from 'pyspark.sql.functions'" . Here the make_date function is directly called on dataframe columns without select expression

 

from pyspark.sql.functions import make_date

df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])

df.select(make_date(df.Y,df.M,df.D).alias("datefield")).show()

 
The error message generated is misleading when it says "cannot  import make_date from pyspark.sql.functions"
Bug: Null at root level of a JSON array causes the parsing failure (w/ permissive mode)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Aug/21 06:35
Affects Version/s: 3.1.2


{code}
scala> spark.read.json(Seq("""[{"a": "str"}, null, {"a": "str"}]""").toDS).collect()
{code}

{code}
...
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (172.30.3.20 executor driver): java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
{code}

Since the mode (by default) is permissive, we shouldn't just fail like above.
Bug: Examples can't run in IDE directly
Status: Open | Priority: Minor | Resolution: 
Created: 05/Aug/21 05:29
Affects Version/s: 3.3.0

I found the examples can't run in IDE(such as Intellij).

For example,  if run `org.apache.spark.examples.sql.JavaUserDefinedScalar` in IDE, the error message as follows:
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
  at org.apache.spark.examples.sql.JavaUserDefinedScalar.main(JavaUserDefinedScalar.java:33)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
  ... 1 more
{code}
Bug: Allow Spark on Kube to operate without polling or watchers
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/21 20:53
Affects Version/s: 3.4.0

Add an option to Spark on Kube to not track the individual executor pods and just assume K8s is doing what it's asked. This would be a developer feature intended for minimizing load on etcd & driver.
Bug: Remove ColumnIO once PARQUET-2050 is released in Parquet 1.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Aug/21 00:41
Affects Version/s: 3.3.0

{{ColumnIO}} doesn't expose methods to get repetition and definition level so Spark has to use a hack. This should be removed once PARQUET-2050 is released.
Bug: Support nested columns in ORC vectorized reader for data source v2
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Aug/21 23:34
Affects Version/s: 3.3.0

We added support of nested columns in ORC vectorized reader for data source v1. Data source v2 and v1 both use same underlying implementation for vectorized reader (OrcColumnVector), so we can support data source v2 as well.
Bug: Expose LogisticRegression.setInitialModel
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Aug/21 15:48
Affects Version/s: 3.3.0

Several Spark ML components already allow setting of an initial model, including KMeans, LogisticRegression, and GaussianMixture. This is useful to begin training from a known reasonably good model.

However, the method in LogisticRegression is private to Spark. I don't see a good reason why it should be as the others in KMeans et al are not.

None of these are exposed in Pyspark, which I don't necessarily want to question or deal with now; there are other places one could arguably set an initial model too, but, here just interested in exposing the existing, tested functionality to callers.
Bug: No longer do file truncate operation before delete a write failed file held by DiskBlockObjectWriter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/21 03:25
Affects Version/s: 3.3.0

We always do a file truncate operation(by DiskBlockObjectWriter.revertPartialWritesAndClose method) before delete a write failed file held by DiskBlockObjectWriter, a typical process is as follows

 
{code:java}
if (!success) {
  // This code path only happens if an exception was thrown above before we set success;
  // close our stuff and let the exception be thrown further
  writer.revertPartialWritesAndClose()
  if (file.exists()) {
    if (!file.delete()) {
      logWarning(s"Error deleting ${file}")
    }
  }
}{code}
 

This truncate operation seems unnecessary, we can add a new method to avoid do it.
Bug: ExecutorMonitor#onBlockUpdated will receive event from driver
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 25/Aug/21 12:18
Affects Version/s: 3.1.2

 When driver broadcast object, it will send the [SparkListenerBlockUpdated|https://github.com/apache/spark/blob/df0ec56723f0b47c3629055fa7a8c63bb4285147/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L228] event. [ExecutorMonitor#onBlockUpdated|https://github.com/apache/spark/blob/df0ec56723f0b47c3629055fa7a8c63bb4285147/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L380] receives and handles the event, in this method, 
 it calls [ensureExecutorIsTracked|https://github.com/apache/spark/blob/df0ec56723f0b47c3629055fa7a8c63bb4285147/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L489]
 to put driver in `executors` variable with `UNKNOWN_RESOURCE_PROFILE_ID`. In my understanding, `ExecutorMonitor` should only monitor executor not driver.  Although this will not cause any problems at the moment because UNKNOWN_RESOURCE_PROFILE_ID will be filtered out, but I think this is a potential risk
Bug: Tasks with Java proxy objects fail to deserialize
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Aug/21 17:05
Affects Version/s: 3.0.3

In JavaSerializer.JavaDeserializationStream we override resolveClass of ObjectInputStream to use the threads' contextClassLoader. However, we do not override resolveProxyClass, which is used when deserializing Java proxy objects, which makes spark use the wrong classloader when deserializing objects, which causes the job to fail with the following exception:

{code}
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, <host>, executor 1): java.lang.ClassNotFoundException: <class>
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at java.base/java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:829)
	at java.base/java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1917)
	...
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
{code}
Bug: Logs should show correct URL of where HistoryServer is started
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Aug/21 21:34
Affects Version/s: 3.3.0

Due to a recent refactoring in the WebUI bind() code, the log message to print the bound host and port information got moved and because of this the info printed is incorrect.

 

Example log - 21/08/05 10:47:38 INFO HistoryServer: Bound HistoryServer to 0.0.0.0, and started at <hostName>:-1

 

Notice above that the port is incorrect
Bug: spark.history.kerberos.principal doesn't take value _HOST
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Aug/21 09:19
Affects Version/s: 3.0.1

spark.history.kerberos.principal doesn't understand value _HOST. 

It says failure to login for principal : spark/_HOST@realm . 

It will be helpful to take _HOST value via config file and change it with current hostname(similar to what Hive does) . This will also help to run SHS on multiple machines without hardcoding principal hostname.  .spark.history.kerberos.principal

 

It require minor change in HistoryServer.scala in initSecurity  method . 

 

Please let me know , if this request make sense , I'll create the PR .
Bug: Use WeakReference over SoftReference in LevelDB to avoid retaining iterator refs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/21 01:09
Affects Version/s: 3.1.2

See discussion at https://github.com/apache/spark/pull/28769#issuecomment-906722390

The SoftReference used to track iterators in LevelDB can prevent them from being GCed early enough, thus finalized and closed. A WeakReference would be more readily collected, and solves an apparent actual issue running a 3.1.x SHS.
Bug: The return value of the Table.capabilities should use EnumSet instead of HashSet
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Aug/21 11:53
Affects Version/s: 3.3.0

The `Table.capabilities` method return a `java.util.Set` of `TableCapability` enumeration type, which is implemented using HashSet now. Such Set objects can be replaced with java.util.EnumSet objects.
Bug: Remove unused listener in HiveThriftServer2AppStatusStore
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Aug/21 07:57
Affects Version/s: 3.1.2

HiveThriftServer2AppStatusStore provides a view of a KVStore with methods that make it easy to query SQL-specific state. 

HiveThriftServer2Listener has no function in this class, we can remove it.
Bug: Replace filter and contains with intersect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Aug/21 03:27
Affects Version/s: 3.3.0

filter on Set can use intersect instead

 

*before*
{code:java}
val set = Set(1, 2)
val others = Set(2, 3)
set.filter(others.contains(_))
{code}
*after*
{code:java}
val set = Set(1, 2)
val others = Set(2, 3)
set.intersect(others)
{code}
Semantic consistency, but it looks simpler
Bug: Improve range split calculation for Kafka Source minPartitions option
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Aug/21 17:17
Affects Version/s: 3.1.2

While the [documentation|https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html] does contain a clear disclaimer,
{quote}Please note that this configuration is like a {{hint}}: the number of Spark tasks will be *approximately* {{minPartitions}}. It can be less or more depending on rounding errors or Kafka partitions that didn't receive any new data.
{quote}
there are cases where the calculated Kafka partition range splits can differ greatly from expectations. For evenly distributed data and most {{minPartitions}} values this would not be a major or commonly encountered concern. However when the distribution of data across partitions is very heavily skewed, somewhat surprising range split calculations can result.

For example, given the following input data:
 * 1 partition containing 10,000 messages
 * 1,000 partitions each containing 1 message

Spark processing code loading from this collection of 1,001 partitions may decide that it would like each task to read no more than 1,000 messages. Consequently, it could specify a {{minPartitions}} value of 1,010 - expecting the single large partition to be split into 10 equal chunks, along with the 1,000 small partitions each having their own task. That is far from what actually occurs. The {{KafkaOffsetRangeCalculator}} algorithm ends up splitting the large partition into 918 chunks of 10 or 11 messages, two orders of magnitude from the desired maximum message count per task and nearly double the number of Spark tasks hinted in the configuration.

Proposing that the {{KafkaOffsetRangeCalculator}}'s range calculation logic be modified to exclude small (i.e. un-split) partitions from the overall proportional distribution math, in order to more reasonably divide the large partitions when they are accompanied by many small partitions, and to provide optimal behavior for cases where a {{minPartitions}} value is deliberately computed based on the volume of data being read.
Bug: Upgrade Jackson to 2.12.5
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/21 08:15
Affects Version/s: 3.2.0

Recently, Jackson 2.12.5 was released and it seems to be expected as the last full patch release for 2.12.x.
This release includes a fix for a regression in jackson-databind introduced in 2.12.3 which Spark 3.2 currently depends on.
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12.5
Bug: Minor UnivariateFeatureSelector API doc improvement
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Aug/21 22:47
Affects Version/s: 3.1.1

* The ScalaDoc and Python doc of the UnivariateFeatureSelector starts with "The user can set ...". The first sentence is the ScalaDoc should always describe what this class is.
* `ANOVATest` is not public. We should just say ANOVA F-test.
Bug: The Cache use weight eviction mechanism has risk of memory leak
Status: In Progress | Priority: Minor | Resolution: 
Created: 26/Aug/21 07:51
Affects Version/s: 3.3.0

In spark, we define Guava Cache in 3 places and use the weight eviction mechanism:
 # ExternalShuffleBlockResolver#shuffleIndexCache
 # RemoteBlockPushResolver#indexCache
 # SharedInMemoryCache#cache

These 3 Guava Cache has risk of memory leakage if configured `maximumWeight` >= 8589934592 (8g) because LocalCache weight eviction does not work when maxSegmentWeight is >= Int.MAX_VALUE( [https://github.com/google/guava/issues/1761|https://github.com/google/guava/issues/1761).])

The UT that can be reproduced this issue is as follows:
{code:java}
@Test
public void testShuffleIndexCacheEvictionBehavior() throws IOException, ExecutionException {
  Map<String, String> config = new HashMap<>();
  String indexCacheSize = "8192m";
  config.put("spark.shuffle.service.index.cache.size", indexCacheSize);
  TransportConf transportConf = new TransportConf("shuffle", new MapConfigProvider(config));
  ExternalShuffleBlockResolver resolver = new ExternalShuffleBlockResolver(transportConf, null);
  resolver.registerExecutor("app0", "exec0", dataContext.createExecutorInfo(SORT_MANAGER));

  LoadingCache<File, ShuffleIndexInformation> shuffleIndexCache = resolver.shuffleIndexCache; // need change access scope of ExternalShuffleBlockResolver.shuffleIndexCache

  // 8g -> 8589934592 bytes
  long maximumWeight = JavaUtils.byteStringAsBytes(indexCacheSize);
  int unitSize = 1048575;
  // CacheBuilder.DEFAULT_CONCURRENCY_LEVEL
  int concurrencyLevel = 4;
  int totalGetCount = 16384;
  // maxCacheCount is 8192
  long maxCacheCount = maximumWeight / concurrencyLevel / unitSize * concurrencyLevel;
  for (int i = 0; i < totalGetCount; i++) {
    File indexFile = new File("shuffle_" + 0 + "_" + i + "_0.index");
    ShuffleIndexInformation indexInfo = Mockito.mock(ShuffleIndexInformation.class);
    Mockito.when(indexInfo.getSize()).thenReturn(unitSize);
    shuffleIndexCache.get(indexFile, () -> indexInfo);
  }

  long totalWeight =
    shuffleIndexCache.asMap().values().stream().mapToLong(ShuffleIndexInformation::getSize).sum();
  long size = shuffleIndexCache.size();
  try{
    Assert.assertTrue(size <= maxCacheCount);
    Assert.assertTrue(totalWeight < maximumWeight);
    fail("The tests code should not enter this line now.");
  } catch (AssertionError error) {
    // The code will enter this branch because LocalCache weight eviction does not work
    // when maxSegmentWeight is >= Int.MAX_VALUE.
    Assert.assertTrue(size > maxCacheCount && size <= totalGetCount);
    Assert.assertTrue(totalWeight > maximumWeight);
  }
}
{code}
 

and from the debug view  we found that there are 2 segment.totalWeight is a negative value

 

!image-2021-08-26-15-56-09-028.png!
Bug:  Add a default value to ORACLE_DOCKER_IMAGE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Aug/21 09:46
Affects Version/s: 3.3.0

Currently the procedure to run the Oracle Integration Suite is based on building the Oracle RDBMS image from the Dockerfiles provided by Oracle.

Recently Oracle has started providing database images, see  https://container-registry.oracle.com

Moreover an Oracle employee is maintaining Oracle XE images that are streamlined for testing at [https://hub.docker.com/r/gvenzl/oracle-xe] and [https://github.com/gvenzl/oci-oracle-xe] This solves the issue that official images are quite large and make testing resource-intensive and slow.

This proposes to document the available options and to introduce a default value for ORACLE_DOCKER_IMAGE
Bug: Move final aggregation in RDD.treeAggregate to executor
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Aug/21 18:02
Affects Version/s: 3.0.0

For the last iteration in RDD.treeAggregate, spark relies on RDD.fold as an implementation detail.
RDD.fold pulls all shuffle partitions to the driver to merge the result.

There are two concerns with this:
a) Shuffle machinery at executors is much more robust/fault tolerant compared to fetching results to driver.
b) Driver is single point of failure in a spark application. When this results in nontrivial increase in memory pressure while pulling partitions to driver or increased memory usage as part of computing the aggregated state (in user code), it can result in driver failures.

For treeAggregate, instead of relying on fold for the last iteration, we should (optionally) do the computation at a single reducer - and fetch the final result to driver.

Additional cost: one extra stage with a single (resulting) partition.
Bug: Handle Stale block fetch failure on the client side by not retrying the requests
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 12/Aug/21 00:33
Affects Version/s: 3.2.0

Similar to SPARK-36378, we need to handle the stale block fetch failures on the client side, although without handling it there won't be any correctness issues. This would help in saving some server side bandwidth unnecessarily serving stale shuffle fetch requests.