av9ash/GitBugs huggingface dataset:
Bug: PrometheusServlet: add "TYPE" comment to exposed metrics
Status: Open | Priority: Major | Resolution: 
Created: 27/Jun/22 14:07
Affects Version/s: 3.0.0

The PrometheusServlet sink does not include the usual comments when exposing the metrics in the prometheus format

e.g. `# TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary`

which prevents some client/integrations that depend on them to assess the metric type to work properly.

For example the AWS cloudwatch agent prometheus plugin attempts to get the metric type from the TYPE comment and considers any metric with no type as unsuported and hence drops it. 

As a result the cloudwatch agent prometheus drops all the metrics exposed by the PrometheusServlet.

[https://github.com/aws/amazon-cloudwatch-agent/blob/1f654cf69c1269073673ba2f636738c556248a31/plugins/inputs/prometheus_scraper/metrics_type_handler.go#L190]

 

This would be solved by adding the TYPE comments to the metrics exposed by the PrometheusServlet sink.

 

_*references:*_
 - [https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-1/]

 - [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala]
 - [https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md]
Bug: Overflow on converting valid Milliseconds to Microseconds
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 21/Jun/22 04:01
Affects Version/s: 3.3.0

 
{code:java}
scala> import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
scala> millisToMicros(Long.MinValue) 
java.lang.ArithmeticException: long overflow
   at java.lang.Math.multiplyExact(Math.java:892)   
    at org.apache.spark.sql.catalyst.util.DateTimeUtils$.millisToMicros(DateTimeUtils.scala:213)
   ... 49 elidedscala> millisToMicros(Long.MinValue)
scala> millisToMicros(Long.MaxValue)
java.lang.ArithmeticException: long overflow
  at java.lang.Math.multiplyExact(Math.java:892)
  at org.apache.spark.sql.catalyst.util.DateTimeUtils$.millisToMicros(DateTimeUtils.scala:213)
  ... 49 elided
{code}
Bug: Fix multiple small bugs in JsonProtocol, impacting StorageLevel and Task/Executor resource requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/22 00:37
Affects Version/s: 1.0.0

While working on SPARK-39489, I spotted three pre-existing bugs in JsonProtocol:
 * {{TaskResourceRequest}} loses precision for values < 0.5. The {{amount}} is a floating point number which is either between 0 and 0.5 or is a positive integer, but the read path assumes it is an integer.
 * {{ExecutorResourceRequest}} integer overflows for values larger than an integer because the write path writes longs but the read path assumes integers.
 * The {{OFF_HEAP}} storage level is not handled properly: the {{offHeap}} field isn't included in the JSON, so this StorageLevel cannot be round-tripped through JSON. This could cause the History Server to display inaccurate "off heap memory used" stats on the executors page.

I'm submitting a separate PR to fix those issues.
Bug: Support ANSI intervals in the percentile functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/22 09:09
Affects Version/s: 3.4.0

Support ANSI intervals in the percentile functions. The feature was disabled by https://issues.apache.org/jira/browse/SPARK-39427 because the functions return double values instead of ANSI intervals. The ticket aims to implement the feature in right way.
Bug: element_at(*, 0) does not return INVALID_ARRAY_INDEX_IN_ELEMENT_AT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/22 23:30
Affects Version/s: 3.3.0

spark-sql> SELECT element_at(array('a', 'b', 'c'), index) FROM VALUES(0), (2) AS T(index);

22/06/09 16:23:07 ERROR SparkSQLDriver: Failed in [SELECT element_at(array('a', 'b', 'c'), index) FROM VALUES(0), (2) AS T(index)]

java.lang.ArrayIndexOutOfBoundsException: SQL array indices start at 1

 at org.apache.spark.sql.errors.QueryExecutionErrors$.sqlArrayIndexNotStartAtOneError(QueryExecutionErrors.scala:1206)

 

This should roll into INVALID_ARRAY_IN_ELEMENT_AT. Makes no sense to make a new error class
Bug: ByteBuffer forget to rewind after get in AvroDeserializer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/22 05:01
Affects Version/s: 3.3.0

{code:java}
case (BYTES, BinaryType) => (updater, ordinal, value) =>
  val bytes = value match {
    case b: ByteBuffer =>
      val bytes = new Array[Byte](b.remaining)
      b.get(bytes)
      // Do not forget to reset the position
      b.rewind()
      bytes
    case b: Array[Byte] => b
    case other => throw new RuntimeException(s"$other is not a valid avro binary.")
  }
  updater.set(ordinal, bytes) {code}
After Avro data is converted to InternalRow, there will be redundant position in ByteBuffer type caused by ByteBuffer#get
Bug: Better error messages for Pandas API on Spark
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 17:12
Affects Version/s: 3.4.0

Currently, some error messages for pandas API on Spark are confusing.

Specifically,
 * when users assume pandas-on-Spark has implemented pandas Scalars 
 * when users pass a ps.Index when creating a DataFrame/Series

Better error messages enhance usability and furthermore, user adoption.

We should improve error messages for Pandas API on Spark.
Bug: Push limit 1 to right side if join type is LeftSemiOrAnti and join condition is empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/22 14:31
Affects Version/s: 3.4.0

Example:
{code:scala}
spark.range(10000).write.saveAsTable("t1")
spark.sql("select * from (select id, '1' as type from t1) where type in (select cast(1 as string) from range(100))").show
{code}
Bug: Datasource v2 scan not pruned in tpcds q10
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Jun/22 09:38
Affects Version/s: 3.2.1



{code:java}
    createAndInitTable("id INT, dep STRING");
    createAndInitTable(
        wideTableName, "id INT, c1 STRING, c2 STRING, c3 STRING, c4 STRING, c5 STRING", null);

    String q1 = String.format("select dep from %s t1 where id > 1 and exists " +
        "(select * from %s t2  where t1.id = t2.id)", tableName, wideTableName);

    Assert.assertEquals("should prune scan", 1, scanCols);
{code}

it looks like select * in subqueries is not fully projected when v2ScanRelationPushdown is applied. after it is later projected the scan wont be changed after that.
Bug: Spark Shuffle Index Cache ignore the weight of index Path
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/22 15:05
Affects Version/s: 2.4.0

We had the same OOMs problem with [SPARK-33206|https://issues.apache.org/jira/browse/SPARK-33206]. This PR fixed the incorrect weight calculation problem when ExternalShuffle caches ShuffleIndexInformation, but we noticed that the key was ignored, of which type is filePath
 
shuffleIndexCache = CacheBuilder.newBuilder()
      .maximumWeight(JavaUtils.byteStringAsBytes(indexCacheSize))
      .weigher((Weigher<String, ShuffleIndexInformation>)
        (filePath, indexInfo) -> indexInfo.getRetainedMemorySize())
      .build(indexCacheLoader);
 
in our case the length of the index path could be greater than 100, e.g. /data/data2/yarn/nm/usercache/hive/appcache/application_1654741161919_1249246/blockmgr-6b0f7db0-7d55-4270-ad3d-42fe70b5694e/37/shuffle_0_1794_0.index
. This is causing a lot of memory usage in jmap dump. Should we consider the key size when calculating the weight in order to get a more accurate result?
Bug: CreateView Command with a window clause query hit a wrong window definition not found issue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/22 20:47
Affects Version/s: 3.3.0

This query will hit a w2 window definition not found in `WindowSubstitute` rule, however this is a bug since the w2 definition is defined in the query.

```
create or replace temporary view test_temp_view as
with step_1 as (
select * , min(a) over w2 as min_a_over_w2 from (select 1 as a, 2 as b, 3 as c) window w2 as (partition by b order by c)) , step_2 as
(
select *, max(e) over w1 as max_a_over_w1
from (select 1 as e, 2 as f, 3 as g)
join step_1 on true
window w1 as (partition by f order by g)
)
select *
from step_2
```


Also we can move the unresolved window expression check from `WindowSubstitute` rule  to `CheckAnalysis` phrase.
Bug: insertIntoHive ExternalTmpPath won't be clear when the app being killed
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/22 10:16
Affects Version/s: 2.4.1

When there is a problem with some types of SparkSql execution (eg: InsertIntoHiveDirCommand and InsertIntoTableDirCommand)
When exiting abnormally, such as being killed by yarn, the .hive-staging directory being written will remain and will not be deleted.

Check the source code to find the specific location here:
org.apache.spark.sql.hive.execution.InsertIntoHiveTable#run

```scala
    val tmpLocation = getExternalTmpPath(sparkSession, hadoopConf, tableLocation)

    try {
      processInsert(sparkSession, externalCatalog, hadoopConf, tableDesc, tmpLocation, child)
    } finally {
      // Attempt to delete the staging directory and the inclusive files. If failed, the files are
      // expected to be dropped at the normal termination of VM since deleteOnExit is used.
      deleteExternalTmpPath(hadoopConf)
    }
```
From spark driver log, I got spark only do shuthook when the application being killed;
I have two questions:
1. The deleteExternalTmpPath method in finally has no effect when the process is killed
2. fs.deleteOnExit(dir) According to the annotation, the data will be cleaned up when the jvm is destroyed

TmpFixed like this:
Bug: Store error message  parameters as Map instead of Array
Status: Open | Priority: Major | Resolution: 
Created: 22/Jun/22 18:20
Affects Version/s: 3.3.1

Right now when we raise a SparkException we pass an array of argument which are assigned to the message parameters by position. This has several downsides:
1. It makes hit hard to later localize (or rework) the messages since that may shuffle position.
2. There could be an accidental mismatch when writing code which is not detected in QA.

3. Sometimes we want to use the same parameter multiple times in a message. Repeating it as an argument seems silly. 

All of these problems go away when we use a map aligning parameters and arguments. We already do this for CheckError.
Bug: to_date function is returning incorrect value
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 20/Jun/22 16:29
Affects Version/s: 3.2.1

Hi,

 

I have a dataframe which has a column containing dates in string format. Now while converting this to date type using to_date , it's giving incorrect date format values. Following is the example code.

 

 

df = spark.createDataFrame(
    [("11/25/1991",), ("1/2/1991",), ("11/30/1991",)], 
    ['date_str']
)

 

spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")

 

df = (df
                 .withColumn('new_date'
                             ,to_date(col('date_str'),'mm/dd/yyyy')))

display(df)

 

 

In the above dataframe we get the date converted correctly for the 2nd row but for 1st and 3rd row we are getting incorrect dates post conversion.

 

 

Could you please look into this issue?

 

Thanks,

Sridhar
Bug: [Yarn] Diagnostics of yarn UI did not display the exception of driver when driver exit before regiserAM
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Jun/22 10:22
Affects Version/s: 3.3.0

If commit a job in yarn cluster mode and driver exited before registerAM，Diagnostics of yarn UI did not show the exception that was throwed by driver .Yarn UI only show :

Application application_xxx failed 1 times (global limit =10; local limit is =1) due to AM Container for appattempt_xxx_000001 exited with exitCode: 13

 

User must view spark log to find the real reason.for example,spark log shows 
{code:java}
2022-06-21,17:58:28,273 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.lang.ArithmeticException: / by zero
java.lang.ArithmeticException: / by zero
	at org.examples.appErrorDemo3$.main(appErrorDemo3.scala:10)
	at org.examples.appErrorDemo3.main(appErrorDemo3.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:736) {code}
 

The reason of this issue is that if driver would not call unregisterAM exited before registerAM ，then yarn UI could not show the real diagnostic information.
Bug: Fix `MultiIndex.value_counts()` when Arrow Execution is enabled
Status: Open | Priority: Major | Resolution: 
Created: 21/Jun/22 22:06
Affects Version/s: 3.4.0

 

When Arrow Execution is enabled,
{code:java}
>>> spark.conf.get("spark.sql.execution.arrow.pyspark.enabled")
'true'
>>> ps.MultiIndex.from_arrays([[1,2], ['a','b']]).value_counts()
{'__index_level_0__': 1, '__index_level_1__': 'a'}    1
{'__index_level_0__': 2, '__index_level_1__': 'b'}    1
dtype: int64
{code}
When Arrow Execution is disabled,
{code:java}
>>> spark.conf.get("spark.sql.execution.arrow.pyspark.enabled")
'false'
>>> ps.MultiIndex.from_arrays([[1,2], ['a','b']]).value_counts()
(1, a)    1
(2, b)    1
dtype: int64 {code}
Notice how indexes of their results are different.

Especially, `value_counts` returns an Index (rather than a MultiIndex), under the hood, a Spark column of StructType (rather than multiple Spark columns), so when Arrow Execution is enabled, Arrow converts the StructType column to a dictionary, where we expect a tuple instead.
Bug: setPredictionCol for OneVsRest does not persist when saving model to disk
Status: Open | Priority: Major | Resolution: 
Created: 21/Jun/22 12:32
Affects Version/s: 3.0.0

The naming of rawPredcitionCol in OneVsRest does not persist after saving and loading a trained model. This becomes an issue when I try to stack multiple One Vs Rest models in a pipeline. Code example below. 
{code:java}
from pyspark.ml.classification import LinearSVC, OneVsRest, OneVsRestModel

data_path = "/sample_multiclass_classification_data.txt"
df = spark.read.format("libsvm").load(data_path)
lr = LinearSVC(regParam=0.01)

# set the name of rawPrediction column
ovr = OneVsRest(classifier=lr, rawPredictionCol = 'raw_prediction')
print(ovr.getRawPredictionCol())

model = ovr.fit(df)model_path = 'temp' + "/ovr_model"

# save and read back in
model.write().overwrite().save(model_path)
model2 = OneVsRestModel.load(model_path)
model2.getRawPredictionCol()

Output:
raw_prediction
'rawPrediction' {code}
Bug: Show "Name" in Spark master UI for drivers
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/22 09:18
Affects Version/s: 3.2.1

For the applications in Spark master in standalone cluster, the "Name" (spark.app.name) shows for running and finished applications, but for driver, Spark master UI only shows main class name, it's hard to map the class name to application name when there are lots of applications and drivers (in some cases, the class name would be same ,but the application name would be difference in a shared cluster), could you show "Name" (spark.app.name) also for Drivers in Spark master UI?
Bug: Parquet bit-packing de/encode optimization
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Jun/22 14:45
Affects Version/s: 3.3.0

Current Spark use Parquet-mr as parquet reader/writer library, but the built-in bit-packing en/decode is not efficient enough. 

Our optimization for Parquet bit-packing en/decode with jdk.incubator.vector in Open JDK18 brings prominent performance improvement.

Due to Vector API is added to OpenJDK since 16, So this optimization request JDK16 or higher.

*Below are our test results*

Functional test is based on open-source parquet-mr Bit-pack decoding function: *_public final void unpack8Values(final byte[] in, final int inPos, final int[] out, final int outPos)_* __

compared with our implementation with vector API *_public final void unpack8Values_vec(final byte[] in, final int inPos, final int[] out, final int outPos)_*

We tested 10 pairs (open source parquet bit unpacking vs ours optimized vectorized SIMD implementation) decode function with bit width=\{1,2,3,4,5,6,7,8,9,10}, below are test results:

!image-2022-06-15-22-50-12-759.png|width=513,height=259!

 

We integrated our bit-packing decode implementation into parquet-mr, tested the parquet batch reader ability from Spark VectorizedParquetRecordReader which get parquet column data by the batch way. We construct parquet file with different row count and column count, the column data type is Int32, the maximum int value is 127 which satisfies bit pack encode with bit width=7,   the count of the row is from 10k to 100 million and the count of the column is from 1 to 4.

!image-2022-06-15-22-52-56-792.png|width=328,height=167!

!image-2022-06-15-22-53-35-937.png|width=354,height=175!

!image-2022-06-15-22-54-07-288.png|width=352,height=173!
Bug: Leverage the natural partitioning and ordering of MonotonicallyIncreasingID
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Jun/22 11:04
Affects Version/s: 3.4.0

In Pandas-API-on-Spark:

1, *MonotonicallyIncreasingID* and *AttachDistributedSequence* are widely used as the indexing type for big tables, they naturally have _RangePartitioning_ and _Ascending Ordering._

2, _Sorting by index_ is a common scenario, such as:

     2.1 internally, {color:#00875a}_Window.orderBy(NATURAL_ORDER_COLUMN_NAME)_{color} is widely used in GroupBy, Rolling, Expanding, EWM, etc;

     2.2 the  {color:#00875a}_sort_index()_{color}   should be a high frequently used function for end users;{color}{color}

 

So we should make *MonotonicallyIncreasingID* and *AttachDistributedSequence* provide the _RangePartitioning_ and _Ascending Ordering, so that the optimizer can leverage them._
Bug: No-null object invoke equal to avoid NPE
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jun/22 08:36
Affects Version/s: 3.2.1

Since {{NullPointerException}} can possibly be thrown while calling the _equals_ method of {{{}Object{}}}, _equals_ should be invoked by a constant or an object that is definitely not {_}null{_}.
{quote}Positive example: {{"test".equals(object);}}
Counter example: {{object.equals("test");}}
{quote}
Bug: Support `ipFamilyPolicy` and `ipFamilies` in Driver Service
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/22 05:13
Affects Version/s: 3.4.0

K8s IPv4/IPv6 dual-stack Feature reached `Stable` stage at v1.23.
 - [https://kubernetes.io/docs/concepts/services-networking/dual-stack/]
 -- v1.16 [alpha]
 -- v1.21 [beta]
 -- v1.23 [stable]

To support IPv6-only environment, we need to control this features.
Bug: Log4j version upgrade to 2.17.2
Status: Resolved | Priority: Major | Resolution: Done
Created: 14/Jun/22 09:36
Affects Version/s: 3.2.1

Hi Team, 

There were talks about upgrading log4j to latest version available as part of security fix.

Wanted to know, if it is already upgraded.

 

Note: We are using  below dependencies,

 
<dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_2.12</artifactId>
          <version>3.0.1</version>
</dependency>
<dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-sql_2.12</artifactId>
          <version>3.0.1</version>
</dependency>
Kindly let us know when the log4j upgrade will be available for users ?
Bug: NullPointerException in SparkRackResolver 
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Jun/22 22:22
Affects Version/s: 2.4.8

When the underlying DNSToSwitchMapping's resolver returns a null we get a NullPointerException:

{noformat}
22/05/24 08:12:12 980 ERROR DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting down SparkContext
java.lang.NullPointerException
	at scala.collection.convert.Wrappers$JListWrapper.isEmpty(Wrappers.scala:87)
	at org.apache.spark.deploy.yarn.SparkRackResolver.coreResolve(SparkRackResolver.scala:76)
{noformat}
Bug: ExternalShuffleService port is extracted from the confs everytime a block manager gets registered
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 14/Jun/22 20:47
Affects Version/s: 3.2.0

We are extracting the shuffle service port from the spark and hadoop configuration with every blockmanager registeration. This could be avoided by using the cached external service port. 

Also, we should avoid calling addMerger when push-based shuffle is disabled.
Bug: Relax AliasAwareOutputExpression to support alias with expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/22 07:09
Affects Version/s: 3.4.0

We will pull out complex join keys from grouping expressions, so the project can hold a alias with expression. Unfortunately we may lose the output partitioning since the current AliasAwareOutputExpression only support preserve the alias with attribute.

For example, the follow query will introduce three exchanges instead of two.
{code:java}
SELECT c1 + 1, count(*)
FROM t1
JOIN t2 ON c1 + 1 = c2
GROUP BY c1 + 1{code}
Bug: local*HostName* methods should support IPv6
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/22 18:28
Affects Version/s: 3.4.0


{code:java}
➜  ./bin/spark-shell
22/06/09 14:52:35 WARN Utils: Your hostname, DBs-Mac-mini-2.local resolves to a loopback address: 127.0.0.1; using 2600:1700:1151:11ef:0:0:0:2000 instead (on interface en1)
22/06/09 14:52:35 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/06/09 14:52:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/09 14:52:44 ERROR SparkContext: Error initializing SparkContext.
java.lang.AssertionError: assertion failed: Expected hostname or IPv6 IP enclosed in [] but got 2600:1700:1151:11ef:0:0:0:2000
	at scala.Predef$.assert(Predef.scala:223) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.util.Utils$.checkHost(Utils.scala:1110) ~[spark-core_2.12-3.2.0.jar:3.2.0.37]
	at org.apache.spark.executor.Executor.<init>(Executor.scala:89) ~[spark-core_2.12-3.2.0.jar:3.2.0.37]
	at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:64) ~[spark-core_2.12-3.2.0.jar:3.2.0]
	at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:132) ~[spark-core_2.12-3.2.0.jar:3.2.0]
{code}
Bug: graph.connectedComponents(maxIterations) get ArrayIndexOutOfBoundsException: -1
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Jun/22 04:52
Affects Version/s: 2.4.3

val graph = Graph(vertices, edges).partitionBy(PartitionStrategy.RandomVertexCut)

The following exceptions are reported during execution，There is no problem when the data volume is small, but an error will be reported when the data volume is large
h1. java.lang.ArrayIndexOutOfBoundsException: -1