av9ash/GitBugs huggingface dataset:
Bug: `Logging plan changes for execution` test failed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/23 03:12
Affects Version/s: 3.3.4

run {{build/sbt clean "sql/test" -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest,org.apache.spark.tags.SlowSQLTest}}

{{}}
{code:java}
2023-06-15T19:58:34.4105460Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32mQueryExecutionSuite:�[0m�[0m
2023-06-15T19:58:34.5395268Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping query execution info to a file (77 milliseconds)�[0m�[0m
2023-06-15T19:58:34.5856902Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping query execution info to an existing file (49 milliseconds)�[0m�[0m
2023-06-15T19:58:34.6099849Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping query execution info to non-existing folder (25 milliseconds)�[0m�[0m
2023-06-15T19:58:34.6136467Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping query execution info by invalid path (4 milliseconds)�[0m�[0m
2023-06-15T19:58:34.6425071Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- dumping query execution info to a file - explainMode=formatted (28 milliseconds)�[0m�[0m
2023-06-15T19:58:34.7084916Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- limit number of fields by sql config (66 milliseconds)�[0m�[0m
2023-06-15T19:58:34.7432299Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- check maximum fields restriction (34 milliseconds)�[0m�[0m
2023-06-15T19:58:34.7554546Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- toString() exception/error handling (11 milliseconds)�[0m�[0m
2023-06-15T19:58:34.7621424Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- SPARK-28346: clone the query plan between different stages (6 milliseconds)�[0m�[0m
2023-06-15T19:58:34.8001412Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m- Logging plan changes for execution *** FAILED *** (12 milliseconds)�[0m�[0m
2023-06-15T19:58:34.8007977Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  testAppender.loggingEvents.exists(((x$10: org.apache.logging.log4j.core.LogEvent) => x$10.getMessage().getFormattedMessage().contains(expectedMsg))) was false (QueryExecutionSuite.scala:232)�[0m�[0m 

{code}
 

but run {{build/sbt "sql/testOnly *QueryExecutionSuite"}} not this issue, need to investigate.
Bug: Session Configs were not getting honored in RDDs
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/23 05:03
Affects Version/s: 3.4.0

When calling SQLConf.get on executors, the configs are read from the local properties on the TaskContext. The local properties are populated driver-side when scheduling the job, using the properties found in sparkContext.localProperties. For RDD actions, local properties were not getting populated.
Bug: Support G1GC in `spark.eventLog.gcMetrics.*` without warning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/23 21:34
Affects Version/s: 3.5.0

{code}
>>> 23/06/23 14:26:53 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors
{code}
Bug: Add SQL functions into Scala, Python and R API
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 01/Jun/23 03:14
Affects Version/s: 3.5.0

See the discussion in dev mailing list (https://lists.apache.org/thread/0tdcfyzxzcv8w46qbgwys2rormhdgyqg).

This is an umbrella JIRA to implement all SQL functions in Scala, Python and R
Bug: Upgrade Arrow to 13.0.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 17:49
Affects Version/s: 4.0.0

According to Apache Arrow release history, 13.0.0 is expected to be released on August 2023.
 - [https://arrow.apache.org/release/]
 -- [12.0.0 (2 May 2023)|https://arrow.apache.org/release/12.0.0.html]
 -- [11.0.0 (26 January 2023)|https://arrow.apache.org/release/11.0.0.html]
 -- [10.0.0 (26 October 2022)|https://arrow.apache.org/release/10.0.0.html]
 -- [9.0.0 (3 August 2022)|https://arrow.apache.org/release/9.0.0.html]
 -- [8.0.0 (6 May 2022)|https://arrow.apache.org/release/8.0.0.html]
Bug: Set io.connectionTimeout/connectionCreationTimeout to zero or negative will cause executor incessantes cons/destructions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 08:03
Affects Version/s: 3.3.2

{code:java}
2023-06-28 14:57:23 CST Bootstrap WARN - Failed to set channel option 'CONNECT_TIMEOUT_MILLIS' with value '-1000' for channel '[id: 0xf4b54a73]'
java.lang.IllegalArgumentException: connectTimeoutMillis : -1000 (expected: >= 0)
	at io.netty.util.internal.ObjectUtil.checkPositiveOrZero(ObjectUtil.java:144) ~[netty-common-4.1.74.Final.jar:4.1.74.Final] {code}
Bug: Spark job submission failed because Xmx string is available on one parameter provided into spark.driver.extraJavaOptions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 08:16
Affects Version/s: 3.3.2

The spark-submit command failed if Xmx string is found on any parameters provided to spark.driver.extraJavaOptions.

For ex. running this spark-submit command line
{code:java}
./bin/spark-submit --class org.apache.spark.examples.SparkPi --conf "spark.driver.extraJavaOptions=-Dtest=Xmx"  examples/jars/spark-examples_2.12-3.4.1.jar 100{code}
failed due to
{code:java}
Error: Not allowed to specify max heap(Xmx) memory settings through java options (was -Dtest=Xmx). Use the corresponding --driver-memory or spark.driver.memory configuration instead.{code}
The check performed in [https://github.com/apache/spark/blob/master/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java#L314] seems to broad
Bug: Even `spark.sql.codegen.factoryMode` is NO_CODEGEN, the WholeStageCodegen also will be generated.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 02:28
Affects Version/s: 3.4.1

The `spark.sql.codegen.factoryMode` is NO_CODEGEN, but Spark always generate WholeStageCodegen plan when set `spark.sql.codegen.wholeStage` to `true`.
Bug: Improve error messages for regular Python UDTFs that return non-tuple values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/23 05:47
Affects Version/s: 3.5.0

Currently, if you have a UDTF like this:
{code:java}
class TestUDTF:
    def eval(self, a: int):
        yield a {code}
and run the UDTF, it will fail with a confusing error message like
{code:java}
Unexpected tuple 1 with StructType {code}
Note this works when arrow is enabled. We should improve error messages for regular UDTFs.
Bug: nesting full outer joins confuses code generator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/23 07:29
Affects Version/s: 3.3.0

We are seeing issues with the code generator when querying java bean encoded data with 2 nested joins.
{code:java}
dsA.join(dsB, seq("id"), "full_outer").join(dsC, seq("id"), "full_outer"); {code}
will generate invalid code in the code generator.  And can depending on the data used generate stack traces like:
{code:java}
 Caused by: java.lang.NullPointerException
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.smj_consumeFullOuterJoinRow_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.wholestagecodegen_findNextJoinRows_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
{code}
Or:
{code:java}
 Caused by: java.lang.AssertionError: index (2) should < 2
        at org.apache.spark.sql.catalyst.expressions.UnsafeRow.assertIndexIsValid(UnsafeRow.java:118)
        at org.apache.spark.sql.catalyst.expressions.UnsafeRow.isNullAt(UnsafeRow.java:315)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.smj_consumeFullOuterJoinRow_0$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage6.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
{code}
When we look at the generated code we see that the code generator seems to be mixing up parameters.  For example:
{code:java}
if (smj_leftOutputRow_0 != null) {                          //<==== null check for wrong/left parameter
  boolean smj_isNull_12 = smj_rightOutputRow_0.isNullAt(1); //<==== causes NPE on right parameter here{code}
It is as if the the nesting of 2 full outer joins is confusing the code generator and as such generating invalid code.

There is one other strange thing.  We found this issue when using data sets which were using the java bean encoder.  We tried to reproduce this in the spark shell or using scala case classes but were unable to do so. 

We made a reproduction scenario as unit tests (one for each of the stacktrace above) on the spark code base and made it available as a [pull request|https://github.com/apache/spark/pull/41688] to this case.
Bug: Positional parameters in sql()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/23 11:02
Affects Version/s: 3.5.0

As a follow-up to the parameterized query we added recently, we’d like to support positional parameters. This is part of the SQL standard and JDBC/ODBC protocol.

Example: update COFFEES set TOTAL = TOTAL + ? where COF_NAME = ?

Note that positional and named param marker cannot be used in the same query.
Bug: Customize diff log in assertDataFrameEqual error message format
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/23 18:10
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Add support for Java UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/23 13:04
Affects Version/s: 3.5.0

API: 
{code:java}
def udf(f: UDF0[_], returnType: DataType): UserDefinedFunction{code}
For 0 - 10 arguments, [reference|https://github.com/apache/spark/blob/747db6675da86e79d04e2fcc531b3b72b22ebf04/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L5624-L5780]
Bug: Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/23 02:30
Affects Version/s: 3.4.0

 
I have a spark application that is deployed using k8s and it is of version 3.3.2 Recently there were some vulneabilities in spark 3.3.2

I changed my dockerfile to download 3.4.0 instead of 3.3.2 and also my application jar is built on spark 3.4.0

However while deploying, I get this error

        

*{{Exception in thread "main" java.nio.file.NoSuchFileException: <path>/spark-assembly-1.0.jar}}*

 

I have this in deployment.yaml of the app

 

*mainApplicationFile: "local:///<path>/spark-assembly-1.0.jar"*

 

 

 

 

and I have not changed anything related to that. I see that some code has changed in spark 3.4.0 core's source code regarding jar location.

Has it really changed the functionality ? Is there anyone who is facing same issue as me ? Should the path be specified in a different way?
Bug: Add error class for the case when loading state from DFS fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 20:14
Affects Version/s: 3.2.0

This is part of [https://github.com/apache/spark/pull/41705.]

Wrap the exception during the loading state, to assign error class properly. With assigning error class, we can classify the errors which help us to determine what errors customers are struggling much. StateStoreProvider.getStore() & StateStoreProvider.getReadStore() is the entry point.

This ticket also covers failedToReadDeltaFileError and failedToReadSnapshotFileError from [https://issues.apache.org/jira/browse/SPARK-36305|http://example.com/].
Bug: Support Python UDTFs with empty return values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/23 05:55
Affects Version/s: 3.5.0

Support UDTFs with empty returns, for example:
{code:java}
@udtf(returnType="a: int")
class TestUDTF:
    def eval(self, a: int):
        ... {code}
Currently, this will fail with the exception 
{code:java}
TypeError: 'NoneType' object is not iterable {code}
Another example
{code:java}
class TestUDTF:
    def eval(self, a: int):
        yield {code}
This will fail with the exceptions
{code:java}
java.lang.NullPointerException {code}
Note, arrow-optimized UDTFs already support this. This error only occurs with regular Python UDTFs.
Bug: Add assertDataFrameEqual util function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/23 22:03
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Make assertSchemaEqual API public
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/23 18:02
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Add PySparkTestBase unit test class
Status: Open | Priority: Major | Resolution: 
Created: 14/Jun/23 22:06
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Upgrade netty dependencies to 4.1.94.Final due to CVE-2023-34462
Status: Open | Priority: Major | Resolution: 
Created: 27/Jun/23 15:47
Affects Version/s: 1.4.1

Hi,

On the Apache Arrow project we have noticed that our nightly integration tests with spark started failing lately. With some investigation I've noticed that we are defining a different version of the Java netty dependencies. We upgraded to 4.1.94.Final due to the CVE on the title: [https://github.com/advisories/GHSA-6mjq-h674-j845]

Our PR upgrading the version: [https://github.com/apache/arrow/issues/36209]

I have opened  an issue on the Apache Arrow repository to try and fix something else on our side but I was wondering if you would want to update the version to solve the CVE.

 

Thanks

Raúl
Bug: Allow custom precision for fp approx equality
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/23 18:03
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Implement call_udf function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/23 12:46
Affects Version/s: 3.5.0

Reference:[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L5824-L5842]

API: 
{noformat}
def call_udf(udfName: String, cols: Column*): Column{noformat}
Bug: Implement UDFRegistration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/23 12:22
Affects Version/s: 3.5.0

Reference file - [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala]

API to be implemented:
 * 
{noformat}
def register(name: String, udf: UserDefinedFunction): UserDefinedFunction{noformat}

 * 
 ** [Reference|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala#L112-L123]

 * 
{noformat}
def register[RT: TypeTag](name: String, func: Function0[RT]): UserDefinedFunction{noformat}

 * 
 ** From [0 to 22 arguments|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala#L125-L642]

 * 
{noformat}
def register(name: String, f: UDF0[_], returnType: DataType): Unit{noformat}

 * 
 ** From [0 to 22 arguments|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala#L735-L1076]

 

We currently do not support UDAFs so the relevant UDAF APIs may be skipped as well as the python/pyspark (in the context of the scala client) related APIs.
Bug: Callback prior to execution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/23 17:10
Affects Version/s: 3.5.0

Commands are eagerly executed after analysis phase, while other queries are executed after planning planning. Users of Spark need to understand time spent prior to execution. Currently, they need to understand the difference between these 2 modes. Add a callback after query planning is completed that can be used for such use.
Bug: Cancel Spark jobs not only by a single "jobgroup", but allow multiple "job tags"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 15:23
Affects Version/s: 3.5.0

Currently, the only way to cancel running Spark Jobs is by using SparkContext.cancelJobGroup, using a job group name that was previously set using SparkContext.setJobGroup. This is problematic if multiple different parts of the system want to do cancellation, and set their own ids.

For example, [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L133] sets it's own job group, which may override job group set by user. This way, if user cancels the job group they set, it will not cancel these broadcast jobs launches from within their jobs...

As a solution, consider adding SparkContext.addJobTag / SparkContext.removeJobTag, which would allow to have multiple "tags" on the jobs, and introduce SparkContext.cancelJobsByTag to allow more flexible cancelling of jobs.
Bug: Remove deprecated API usage in HiveShim.scala
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 14/Jun/23 18:05
Affects Version/s: 3.4.0

Spark's HiveShim.scala calls this particular method in Hive :
createPartitionMethod.invoke(
hive,
table,
spec,
location,
params, // partParams
null, // inputFormat
null, // outputFormat
-1: JInteger, // numBuckets
null, // cols
null, // serializationLib
null, // serdeParams
null, // bucketCols
null) // sortCols
}
 
We do not have any such implementation of createPartition in Hive. We only have this definition :
public Partition createPartition(Table tbl, Map<String, String> partSpec) throws HiveException {
    try

{       org.apache.hadoop.hive.metastore.api.Partition part =           Partition.createMetaPartitionObject(tbl, partSpec, null);       AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf, tbl);       part.setWriteId(tableSnapshot != null ? tableSnapshot.getWriteId() : 0);       return new Partition(tbl, getMSC().add_partition(part));     }

catch (Exception e)

{       LOG.error(StringUtils.stringifyException(e));       throw new HiveException(e);     }

  }
*The 12 parameter implementation was removed in HIVE-5951*

 

The issue is that this 12 parameter implementation of createPartition method was added in Hive-0.12 and then was removed in Hive-0.13. When Hive 0.12 was used in Spark, SPARK-15334 commit in Spark added this 12 parameters implementation. But after Hive migrated to newer APIs somehow this was not changed in Spark OSS and it looks to us like a Bug from the Spark end.

 

We need to migrate to the newest implementation of Hive createPartition method otherwise this flow can break
Bug: Potential for incorrect results or NPE when full outer USING join has null key value
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/23 19:53
Affects Version/s: 3.3.2

The following query produces incorrect results:
{noformat}
create or replace temp view v1 as values (1, 2), (null, 7) as (c1, c2);
create or replace temp view v2 as values (2, 3) as (c1, c2);

select explode(array(c1)) as x
from v1
full outer join v2
using (c1);

-1   <== should be null
1
2
{noformat}
The following query fails with a {{NullPointerException}}:
{noformat}
create or replace temp view v1 as values ('1', 2), (null, 7) as (c1, c2);
create or replace temp view v2 as values ('2', 3) as (c1, c2);

select explode(array(c1)) as x
from v1
full outer join v2
using (c1);

23/06/25 17:06:39 ERROR Executor: Exception in task 0.0 in stage 14.0 (TID 11)
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.generate_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.smj_consumeFullOuterJoinRow_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.wholestagecodegen_findNextJoinRows_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
...
{noformat}
Bug: Client receives zero number of chunks in merge meta response which doesn't trigger fallback to unmerged blocks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/23 16:55
Affects Version/s: 3.2.0

We still see instances of the server returning 0 {{numChunks}} in {{mergedMetaResponse}} which causes the executor to fail with {{ArithmeticException}}. 
{code}
java.lang.ArithmeticException: / by zero
	at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:128)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:1047)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:90)
	at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
{code}
Here the executor doesn't fallback to fetch un-merged blocks and this also doesn't result in a {{FetchFailure}}. So, the application fails.
Bug: Bitmap functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/23 05:12
Affects Version/s: 3.5.0

Implemented bitmap functions. The functions are:
 * {{{}bitmap_bucket_number(){}}}: returns the bucket number for a given input number
 * {{{}bitmap_bit_position(){}}}: returns bit position for a given input number
 * {{{}bitmap_count(){}}}: returns the number of set bits from an input bitmap
 * {{{}bitmap_construct_agg(){}}}: aggregation function that aggregates input bit positions, and creates a bitmap
 * {{{}bitmap_or_agg(){}}}: aggregation function that performs a bitwise OR on all the input bitmaps
Bug: Add JobTag APIs to Pyspark SparkContext
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/23 16:02
Affects Version/s: 3.5.0

Add APIs added in https://issues.apache.org/jira/browse/SPARK-43952 to PySpark:
 * {{SparkContext.addJobTag(tag: String): Unit}}
 * {{SparkContext.removeJobTag(tag: String): Unit}}
 * {{SparkContext.getJobTags(): Set[String]}}
 * {{SparkContext.clearJobTags(): Unit}}
 * {{SparkContext.cancelJobsWithTag(tag: String): Unit}}