av9ash/GitBugs huggingface dataset:
Bug: Unnecessary persist on cumulativeCounts in BinaryClassificationMetrics
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 22/Mar/20 15:32
Affects Version/s: 2.4.4

In mllib.evaluation.BinaryClassificationMetrics, _cumulativeCounts_ is cached in a lazy initialization. But when I run LogisticRegressionSummaryExample as well as ModelSelectionViaCrossValidationExample, I find that cached _cumulativeCounts_ only used by one action during execution. 
So I think it should not be cached in initilization, we can set an extra persist() API in this class, just as that the unpersist() API in BinaryClassificationMetrics releases cached _cumulativeCounts_.
Bug: Offsets are only logged for partitions which had data this causes next batch to read the partitions that were not included from the beginning when using kafka
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 04/Mar/20 15:47
Affects Version/s: 2.4.0

Each batch should either log all offsets for each partition or should scan back across offset logs.

[https://github.com/apache/spark/blob/v2.4.4/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala]

offset log 23615

 
{code:java}
{"myTopic.myTopic.orders":{"2":27531503,"5":27562423,"4":27528794,"1":27514991,"3":27528899,"0":27504949}}%
{code}
 

 

offset log 23616

 
{code:java}
{"myTopic.myTopic.orders":{"1":27515130,"0":27505140}}%
{code}
 

 
{code:java}
/0/03/04 13:49:05 INFO MicroBatchExecution: Resuming at batch 26317 with committed offsets {KafkaV2[Subscribe[myTopic.myTopic.orders]]: {"myTopic.myTopic.orders":{"1":27515130,"0":27505140}}} and available offsets {KafkaV2[Subscribe[myTopic.myTopic.orders]]: {"myTopic.myTopic.orders":{"2":27531625,"5":27562568,"4":27528990,"1":27515131,"3":27529075,"0":27505141}}}commit log: {"myTopic.myTopic.orders":{"1":27515130,"0":27505140}}%0/03/04 13:50:24 INFO KafkaMicroBatchReader: Partitions added: Map(myTopic.myTopic.orders-3 -> 26533520, myTopic.myTopic.orders-2 -> 26533730, myTopic.myTopic.orders-4 -> 26533608, myTopic.myTopic.orders-5 -> 26533486)
20/03/04 13:50:24 WARN KafkaMicroBatchReader: Added partition myTopic.myTopic.orders-3 starts from 26533520 instead of 0. Some data may have been missed.
20/03/04 13:50:24 WARN KafkaMicroBatchReader: Added partition myTopic.myTopic.orders-2 starts from 26533730 instead of 0. Some data may have been missed.
20/03/04 13:50:24 WARN KafkaMicroBatchReader: Added partition myTopic.myTopic.orders-4 starts from 26533608 instead of 0. Some data may have been missed.
20/03/04 13:50:24 WARN KafkaMicroBatchReader: Added partition myTopic.myTopic.orders-5 starts from 26533486 instead of 0. Some data may have been missed.

{code}
Bug: Shared Volume Spark Kubernetes
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 30/Mar/20 09:34
Affects Version/s: 2.4.5

Please change the deployment script for the workers to support one Shared Volume with the driver. E.g. a possible implementation:
https://kubernetes.io/docs/tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/

Atm. If a driver runs in Kubernetes it is not possible to share a volume between the driver and the workers. 
Enabling a shared volume between Driver and Workers would allow the use of sc.textFile(file:///path/file.txt) to be used without any need of HDFS or S3 Storage (Which in fact is a huge benefit)
Bug: Wrap java.lang.Error with an exception for QueryExecutionListener.onFailure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Mar/20 06:35
Affects Version/s: 3.0.0

SPARK-28556 changed the method QueryExecutionListener.onFailure to allow Spark sending java.lang.Error to this method. As this change breaks APIs, we cannot fix branch-2.4.

[~marmbrus] suggested to wrap java.lang.Error with an exception instead to avoid a breaking change. A bonus of this solution is we can also fix the issue (if a query throws java.lang.Error, QueryExecutionListener doesn't get notified) in branch-2.4.
Bug: Spark Cli does not respect hive-site.xml and spark.sql.warehouse.dir
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/20 03:45
Affects Version/s: 3.0.0

In Spark CLI, we create a hive CliSessionState and it does not load the hive-site.xml. So the configurations in hive-site.xml will not take effects like other spark-hive integration apps.

Also, the warehouse directory is not correctly picked. If the `default` database does not exist, the CliSessionState will create one during the first time it talks to the metastore. The `Location` of the default DB will be neither the value of spark.sql.warehousr.dir nor the user-specified value of hive.metastore.warehourse.dir, but the default value of hive.metastore.warehourse.dir which will always be `/user/hive/warehouse`.
Bug: AQE will use the same SubqueryExec even if subqueryReuseEnabled=false
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 20/Mar/20 14:40
Affects Version/s: 3.0.0

In `InsertAdaptiveSparkPlan.buildSubqueryMap`, AQE will skip to compile the subquery with the same exprId. As a result, in PlanAdaptiveSubqueries, it will use the same SubqueryExec for the SubqueryExpression with the same exprId.
Bug: Unify create table syntax to fix ambiguous two different CREATE TABLE syntaxes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 23:23
Affects Version/s: 3.1.0

There's a discussion in dev@ mailing list to point out ambiguous syntaxes for CREATE TABLE DDL. This issue tracks the efforts to resolve the root issue via unifying the create table syntax.

[https://lists.apache.org/thread.html/rf1acfaaa3de2d3129575199c28e7d529d38f2783e7d3c5be2ac8923d%40%3Cdev.spark.apache.org%3E]

We should ensure the new "single" create table syntax is very deterministic to both devs and end users.

This work also includes how to pass the extra CREATE TABLE information(from the Hive tyle CREATE TABLE) to the v2 catalog:

1.  turn the Hive serde info to table properties with `option.` prefix.

2. add a new v2 table option `external` to indicate CREATE EXTERNAL TABLE.
Bug: Upgrade Janino to 3.0.16
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/20 00:48
Affects Version/s: 3.0.0

We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details.
Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly.
Bug: Impossible to change the provider of a table in the HiveMetaStore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 00:49
Affects Version/s: 3.0.0

Currently, it's impossible to alter the datasource of a table in the HiveMetaStore by using alterTable, as the HiveExternalCatalog doesn't change the provider table property during an alterTable command. This is required to support changing table formats when using commands like REPLACE TABLE.
Bug: Parquet predicate pushdown on columns with dots
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/20 19:56
Affects Version/s: 3.1.0

Parquet predicate pushdown on columns with dots is disabled in -SPARK-20364- due to Parquet's APIs don't support it. A new set of APIs is purposed in PARQUET-1809 to generalize the support of nested cols which can address this issue. This implementation will be merged into Spark repo first until we get a new release from Parquet community.
Bug: Upgrade netty-all to 4.1.47.Final
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/20 14:38
Affects Version/s: 2.4.5

Upgrade version of io.netty_netty-all to 4.1.44.Final [CVE-2019-20445|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-20445]
Bug: repartition obeys spark.sql.adaptive.coalescePartitions.initialPartitionNum when spark.sql.adaptive.enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 05:17
Affects Version/s: 3.0.0


{code:scala}
spark.sql("CREATE TABLE spark_31220(id int)")
spark.sql("set spark.sql.adaptive.coalescePartitions.initialPartitionNum=1000")
spark.sql("set spark.sql.adaptive.enabled=true")
{code}

{noformat}
scala> spark.sql("SELECT id from spark_31220 GROUP BY id").explain
== Physical Plan ==
AdaptiveSparkPlan(isFinalPlan=false)
+- HashAggregate(keys=[id#5], functions=[])
   +- Exchange hashpartitioning(id#5, 1000), true, [id=#171]
      +- HashAggregate(keys=[id#5], functions=[])
         +- FileScan parquet default.spark_31220[id#5] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/spark_31220], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>



scala> spark.sql("SELECT id from spark_31220 DISTRIBUTE BY id").explain
== Physical Plan ==
AdaptiveSparkPlan(isFinalPlan=false)
+- Exchange hashpartitioning(id#5, 200), false, [id=#179]
   +- FileScan parquet default.spark_31220[id#5] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/root/opensource/apache-spark/spark-warehouse/spark_31220], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>
{noformat}
Bug: Occasional class not found error in user's Future code using global ExecutionContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/20 02:11
Affects Version/s: 2.4.5

*Problem:*
When running tpc-ds test (https://github.com/databricks/spark-sql-perf), occasionally we see error related to class not found:

2020-02-04 20:00:26,673 ERROR yarn.ApplicationMaster: User class threw exception: scala.ScalaReflectionException: class com.databricks.spark.sql.perf.ExperimentRun in JavaMirror with 
sun.misc.Launcher$AppClassLoader@28ba21f3 of type class sun.misc.Launcher$AppClassLoader with classpath [...] 
and parent being sun.misc.Launcher$ExtClassLoader@3ff5d147 of type class sun.misc.Launcher$ExtClassLoader with classpath [...] 
and parent being primordial classloader with boot classpath [...] not found.

*Root cause:*
Spark driver starts ApplicationMaster in the main thread, which starts a user thread and set MutableURLClassLoader to that thread's ContextClassLoader.
	userClassThread = startUserApplication()

The main thread then setup YarnSchedulerBackend RPC endpoints, which handles these calls using scala Future with the default global ExecutionContext:
    - doRequestTotalExecutors
    - doKillExecutors

If main thread starts a future to handle doKillExecutors() before user thread does then the default thread pool thread's ContextClassLoader would be the default (AppClassLoader). 
If user thread starts a future first then the thread pool thread will have MutableURLClassLoader.

So if user's code uses a future which references a user provided class (only MutableURLClassLoader can load), and before the future if there are executor lost, you will see errors related to class not found.

*Proposed Solution:*
We can potentially solve this problem in one of two ways:
1) Set the same class loader (userClassLoader) to both the main thread and user thread in ApplicationMaster.scala

2) Do not use "ExecutionContext.Implicits.global" in YarnSchedulerBackend
Bug: Add CalendarIntervals division
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 05/Mar/20 15:48
Affects Version/s: 3.1.0

{{CalendarInterval}} should be allowed for division. The {{CalendarInterval}} consists of three time components: {{months}}, {{days}} and {{microseconds}}. The division can only be defined between intervals that have a single non-zero time component, while both intervals have the same non-zero time component. Otherwise the division expression would be ambiguous.

This allows to evaluate the magnitude of {{CalendarInterval}} in SQL expressions:
{code}
Seq((Timestamp.valueOf("2020-02-01 12:00:00"), Timestamp.valueOf("2020-02-01 13:30:25")))
  .toDF("start", "end")
  .withColumn("interval", $"end" - $"start")
  .withColumn("interval [h]", $"interval" / lit("1 hour").cast(CalendarIntervalType))
  .withColumn("rate [€/h]", lit(1.45))
  .withColumn("price [€]", $"interval [h]" * $"rate [€/h]")
  .show(false)
+-------------------+-------------------+-----------------------------+------------------+----------+------------------+
|start              |end                |interval                     |interval [h]      |rate [€/h]|price [€]         |
+-------------------+-------------------+-----------------------------+------------------+----------+------------------+
|2020-02-01 12:00:00|2020-02-01 13:30:25|1 hours 30 minutes 25 seconds|1.5069444444444444|1.45      |2.1850694444444443|
+-------------------+-------------------+-----------------------------+------------------+----------+------------------+
{code}

The currently available approach is

{code}
Seq((Timestamp.valueOf("2020-02-01 12:00:00"), Timestamp.valueOf("2020-02-01 13:30:25")))
  .toDF("start", "end")
  .withColumn("interval [s]", unix_timestamp($"end") - unix_timestamp($"start"))
  .withColumn("interval [h]", $"interval [s]" / 3600)
  .withColumn("rate [€/h]", lit(1.45))
  .withColumn("price [€]", $"interval [h]" * $"rate [€/h]")
  .show(false)
{code}

Going through {{unix_timestamp}} is a hack and it pollutes the SQL query with unrelated semantics (unix timestamp is completely irrelevant for this computation). It is merely there because there is currently no way to access the length of an {{CalendarInterval}}. Dividing an interval by another interval provides means to measure the length in an arbitrary unit (minutes, hours, quarter hours).
Bug: spark-sql fails to parse when contains comment
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/20 03:07
Affects Version/s: 3.0.0

{code:sql}
select
  1,
  -- two
  2;
{code}


{noformat}
spark-sql> select
         >   1,
         >   -- two
         >   2;
Error in query:
mismatched input '<EOF>' expecting {'(', 'ADD', 'AFTER', 'ALL', 'ALTER', 'ANALYZE', 'AND', 'ANTI', 'ANY', 'ARCHIVE', 'ARRAY', 'AS', 'ASC', 'AT', 'AUTHORIZATION', 'BETWEEN', 'BOTH', 'BUCKET', 'BUCKETS', 'BY', 'CACHE', 'CASCADE', 'CASE', 'CAST', 'CHANGE', 'CHECK', 'CLEAR', 'CLUSTER', 'CLUSTERED', 'CODEGEN', 'COLLATE', 'COLLECTION', 'COLUMN', 'COLUMNS', 'COMMENT', 'COMMIT', 'COMPACT', 'COMPACTIONS', 'COMPUTE', 'CONCATENATE', 'CONSTRAINT', 'COST', 'CREATE', 'CROSS', 'CUBE', 'CURRENT', 'CURRENT_DATE', 'CURRENT_TIME', 'CURRENT_TIMESTAMP', 'CURRENT_USER', 'DATA', 'DATABASE', DATABASES, 'DAY', 'DBPROPERTIES', 'DEFINED', 'DELETE', 'DELIMITED', 'DESC', 'DESCRIBE', 'DFS', 'DIRECTORIES', 'DIRECTORY', 'DISTINCT', 'DISTRIBUTE', 'DROP', 'ELSE', 'END', 'ESCAPE', 'ESCAPED', 'EXCEPT', 'EXCHANGE', 'EXISTS', 'EXPLAIN', 'EXPORT', 'EXTENDED', 'EXTERNAL', 'EXTRACT', 'FALSE', 'FETCH', 'FIELDS', 'FILTER', 'FILEFORMAT', 'FIRST', 'FOLLOWING', 'FOR', 'FOREIGN', 'FORMAT', 'FORMATTED', 'FROM', 'FULL', 'FUNCTION', 'FUNCTIONS', 'GLOBAL', 'GRANT', 'GROUP', 'GROUPING', 'HAVING', 'HOUR', 'IF', 'IGNORE', 'IMPORT', 'IN', 'INDEX', 'INDEXES', 'INNER', 'INPATH', 'INPUTFORMAT', 'INSERT', 'INTERSECT', 'INTERVAL', 'INTO', 'IS', 'ITEMS', 'JOIN', 'KEYS', 'LAST', 'LATERAL', 'LAZY', 'LEADING', 'LEFT', 'LIKE', 'LIMIT', 'LINES', 'LIST', 'LOAD', 'LOCAL', 'LOCATION', 'LOCK', 'LOCKS', 'LOGICAL', 'MACRO', 'MAP', 'MATCHED', 'MERGE', 'MINUTE', 'MONTH', 'MSCK', 'NAMESPACE', 'NAMESPACES', 'NATURAL', 'NO', NOT, 'NULL', 'NULLS', 'OF', 'ON', 'ONLY', 'OPTION', 'OPTIONS', 'OR', 'ORDER', 'OUT', 'OUTER', 'OUTPUTFORMAT', 'OVER', 'OVERLAPS', 'OVERLAY', 'OVERWRITE', 'PARTITION', 'PARTITIONED', 'PARTITIONS', 'PERCENT', 'PIVOT', 'PLACING', 'POSITION', 'PRECEDING', 'PRIMARY', 'PRINCIPALS', 'PROPERTIES', 'PURGE', 'QUERY', 'RANGE', 'RECORDREADER', 'RECORDWRITER', 'RECOVER', 'REDUCE', 'REFERENCES', 'REFRESH', 'RENAME', 'REPAIR', 'REPLACE', 'RESET', 'RESTRICT', 'REVOKE', 'RIGHT', RLIKE, 'ROLE', 'ROLES', 'ROLLBACK', 'ROLLUP', 'ROW', 'ROWS', 'SCHEMA', 'SECOND', 'SELECT', 'SEMI', 'SEPARATED', 'SERDE', 'SERDEPROPERTIES', 'SESSION_USER', 'SET', 'MINUS', 'SETS', 'SHOW', 'SKEWED', 'SOME', 'SORT', 'SORTED', 'START', 'STATISTICS', 'STORED', 'STRATIFY', 'STRUCT', 'SUBSTR', 'SUBSTRING', 'TABLE', 'TABLES', 'TABLESAMPLE', 'TBLPROPERTIES', TEMPORARY, 'TERMINATED', 'THEN', 'TO', 'TOUCH', 'TRAILING', 'TRANSACTION', 'TRANSACTIONS', 'TRANSFORM', 'TRIM', 'TRUE', 'TRUNCATE', 'TYPE', 'UNARCHIVE', 'UNBOUNDED', 'UNCACHE', 'UNION', 'UNIQUE', 'UNKNOWN', 'UNLOCK', 'UNSET', 'UPDATE', 'USE', 'USER', 'USING', 'VALUES', 'VIEW', 'WHEN', 'WHERE', 'WINDOW', 'WITH', 'YEAR', '+', '-', '*', 'DIV', '~', STRING, BIGINT_LITERAL, SMALLINT_LITERAL, TINYINT_LITERAL, INTEGER_VALUE, EXPONENT_VALUE, DECIMAL_VALUE, DOUBLE_LITERAL, BIGDECIMAL_LITERAL, IDENTIFIER, BACKQUOTED_IDENTIFIER}(line 3, pos 2)

== SQL ==
select
  1,

--^^^
{noformat}
Bug: Separate connection timeout and idle timeout for netty
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Mar/20 04:01
Affects Version/s: 3.1.0

io.connectionTimeout only used for connection timeout for connection setup while io.idleTimeout is used to control how long to kill the connection if it seems to be idle([https://github.com/apache/spark/pull/5584])

 

These 2 timeouts could be quite different. connectiontimeout could be short to help fast fail the connection related problem when meet some high load/unstable nodes. While idle timeout may be related to business function performance which is more complicated.
Bug: Fast fail the connection while last shuffle connection failed in the last retry IO wait 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/20 02:12
Affects Version/s: 3.1.0

When reading shuffle data, maybe several fetch request sent to a same shuffle server.
There is a client pool, and these request may share the same client.
When the shuffle server is busy, it may cause the request connection timeout.
For example: there are two request connection, rc1 and rc2.
Especially, the io.numConnectionsPerPeer is 1 and connection timeout is 2 minutes.

1: rc1 hold the client lock, it timeout after 2 minutes.
2: rc2 hold the client lock, it timeout after 2 minutes.
3: rc1 start the second retry, hold lock and timeout after 2 minutes.
4: rc2 start the second retry, hold lock and timeout after 2 minutes.
5: rc1 start the third retry, hold lock and timeout after 2 minutes.
6: rc2 start the third retry, hold lock and timeout after 2 minutes.
It wastes lots of time.
Bug: YarnShuffleService doesn't close idle netty channel
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 03:27
Affects Version/s: 2.4.5

Recently, we find our YarnShuffleService has a lot of [half-open connections|https://blog.stephencleary.com/2009/05/detection-of-half-open-dropped.html] where shuffle servers' connections are active while clients have already closed. 

For example, from server's `ss -nt sport = :7337` output we have
{code:java}
ESTAB 0 0 server:7337 client:port

{code}
However, on client `ss -nt dport =: 7337 | grep server` would return nothing.

Looking at the code,  `YarnShuffleService` creates a `TransportContext` with `closeIdleConnections` set to false.
{code:java}
public class YarnShuffleService extends AuxiliaryService {
  ...
  @Override  protected void serviceInit(Configuration conf) throws Exception { 
    ...     
    transportContext = new TransportContext(transportConf, blockHandler); 
    ...
  }
  ...
}

public class TransportContext implements Closeable {
  ...

  public TransportContext(TransportConf conf, RpcHandler rpcHandler) {       
    this(conf, rpcHandler, false, false);  
  }
  public TransportContext(TransportConf conf, RpcHandler rpcHandler, boolean closeIdleConnections) {    
    this(conf, rpcHandler, closeIdleConnections, false);  
  }
  ...
}{code}
Hence, it's possible the channel  may never get closed at server side if the server misses the event that the client has closed it.

I find that parameter is true for `ExternalShuffleService`.

Is there any reason for the difference here ?  Can we enable closeIdleConnections in YarnShuffleService or at least add a configuration to enable it ?
Bug: Use Minio instead of Ceph in K8S DepsTestsSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 06:11
Affects Version/s: 3.0.0

`DepsTestsSuite` is using `ceph` for S3 storage. However, it's not robust on `minikube` version. Also, the image size is almost 1GB. 
{code}
ceph/daemon                               v4.0.3-stable-4.0-nautilus-centos-7-x86_64   a6a05ccdf924        6 months ago        852MB
ceph/daemon                               v4.0.11-stable-4.0-nautilus-centos-7         87f695550d8e        12 hours ago        901MB
{code}

{code}
$ minikube version
minikube version: v1.8.2

$ minikube -p minikube docker-env | source

$ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.3-stable-4.0-nautilus-centos-7-x86_64 /bin/sh
2020-03-25 04:26:21  /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings

$ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.11-stable-4.0-nautilus-centos-7 /bin/sh
2020-03-25 04:20:30  /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings
{code}
Bug: Add "-XX:ActiveProcessorCount" to Spark driver and executor in Yarn mode
Status: Closed | Priority: Major | Resolution: Invalid
Created: 04/Mar/20 01:23
Affects Version/s: 2.4.5

When starting Spark driver and executors on Yarn cluster, the JVM process can discover all CPU cores on the system and set thread-pool or GC threads based on that value. We should limit what the JVM sees for the number of cores set by the user (spark.driver.cores or spark.executor.cores) by "-XX:ActiveProcessorCount", which was introduced in Java 8u191.

Especially in running Spark on Yarn inside Kubernetes container, the number of CPU cores discovered sometimes is 1, which means it always use 1 thread in the default thread pool, or GC threads.
Bug: rand() function documentation suggests an inclusive upper bound of 1.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Mar/20 17:28
Affects Version/s: 2.4.5

 The rand() function in PySpark, Spark, and R is documented as drawing from U[0.0, 1.0]. This suggests an inclusive upper bound, and can be confusing (i.e for a distribution written as `X ~ U(a, b)`, x can be a or b, so writing `U[0.0, 1.0]` suggests the value returned could include 1.0). The function itself uses Rand(), which is [documented |#L71] as having a result in the range [0, 1).