av9ash/GitBugs huggingface dataset:
Bug: Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 15/Dec/20 12:04
Affects Version/s: 3.0.1

FsHistoryProvider#checkForLogs already has FileStatus when constructing SingleFileEventLogFileReader, and there is no need to get the FileStatus again when SingleFileEventLogFileReader#fileSizeForLastIndex.
This can reduce a lot of rpc calls and improve the speed of the history server.
Bug: Three cases always fail in hive-thriftserver module
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Dec/20 06:51
Affects Version/s: 3.2.0

Seems the following tests always failed both in Jenkins and Github Action:
 * org.apache.spark.sql.hive.thriftserver.HiveThriftHttpServerSuite.JDBC query execution
 * org.apache.spark.sql.hive.thriftserver.HiveThriftHttpServerSuite.Checks Hive version
 * org.apache.spark.sql.hive.thriftserver.HiveThriftHttpServerSuite.SPARK-24829 Checks cast as float

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132189/testReport/]

[https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132403/testReport/]
Bug: Add an option in AQE to mitigate skew even if it causes an new shuffle
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/20 22:08
Affects Version/s: 3.0.0

Currently {{OptimizeSkewedJoin}} will not apply if skew mitigation causes a new shuffle.
There are situations where it's better to mitigate skew even if it means a new shuffle is added, for example if the join outputs small amount of data.

As a first step I propose adding a SQLConf option to enable this.  

I'll open a PR shortly to get feedback on the approach.
Bug: Avoid the getSimpleMessage of AnalysisException adds semicolon repeatedly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/20 07:04
Affects Version/s: 3.2.0

The current getSimpleMessage of AnalysisException may adds semicolon repeatedly. There show an example below:

{code:java}
select decode()
{code}
The output will be:

{code:java}
org.apache.spark.sql.AnalysisException
Invalid number of arguments for function decode. Expected: 2; Found: 0;; line 1 pos 7
{code}
Bug: Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Dec/20 08:56
Affects Version/s: 3.0.0

When partitions are tracked by the catalog, that will compute all custom partition locations, especially when dynamic partitions, and the field staticPartitions is empty.
 The poor performance of the method listPartitions results in a long period of no response at the Driver.

When read 12253 partitions, the method getPartitionsByNames takes 2 seconds, and the getPartitions takes 457 seconds, nearly 8 minutes

!image-2020-12-02-16-57-43-619.png|width=783,height=54!
Bug: How can we submit or initiate multiple spark application with single or few JVM
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/20 03:29
Affects Version/s: 2.4.5

How can we have single JVM or few JVM process submit multiple application to cluster.

It is observed that each spark-submit opens upto 400 JARS of >1GB size and creates  _spark_conf_XXXX.zip in /tmp  and copy under application specific .staging directory.    When run concurrently for # of JVMs that can be supported in a server is limited and 100% CPU during job submission and  until client java processes start exiting.

Initially we thought creating zip files and distributing this to hdfs for each application is the source of issue. However reducing the size of zipfile by 50% also we didn't see much difference and indicates the main source of issue is number of JAVA process on client side.

Direct impact is any submission with concurrency >40 (#of hyperthreaded cores) leads to failure and CPU overload on GW. Tried Livy, however noticed, in the background this solution also does a spark-submit and same problem persists and getting "response code 404" and observe the same CPU overload on server running livy. The concurrency is due to mini-batches over REST and expecting and try to support 2000+ concurrent requests as long as we have the resource to support in the cluster. For this spark-submit is the major bottleneck because of the explained situation. For JARS submission, we have more than one work-around (1.pre-distribute the jars to a specified folder and refer local keyword or 2) stage the JARS in a HDFS location and specify HDFS reference thus no file-copy per application).

Is there a way to create a service/services that will stay running and submit jobs to cluster. For running application in Client mode make sense to open 400+ jars, however just for sumibtting the application to cluster we could have a simple/lite process that runs as service.

Regards,
-Ramesh
Bug: Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/20 20:01
Affects Version/s: 3.2.0

In Yarn cluster modes, the passed files are able to be accessed in the current working directory. Looks like this is not the case in Kubernates cluset mode.

By doing this, users can, for example, leverage PEX to manage Python dependences in Apache Spark:

{code}
pex pyspark==3.0.1 pyarrow==0.15.1 pandas==0.25.3 -o myarchive.pex
PYSPARK_PYTHON=./myarchive.pex spark-submit --files myarchive.pex
{code}

See also https://github.com/apache/spark/pull/30735/files#r540935585.
Bug: spark sql datasourceV2 support   kafka table 
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/20 09:59
Affects Version/s: 3.0.1

ddl sql like below:

 

{{CREATE Stream  kafkatable}}

{{USING org.apache.spark.sql.kafkaOPTIONS (}}

{{  topic "topic",}}

{{......}}

{{}}

{{)}}
Bug: Build and Run Spark on Java 17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/20 01:39
Affects Version/s: 3.3.0

Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 17.

||Version||Release Date||
|Java 17 (LTS)|September 2021|


Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along with the release branch cut.
- https://spark.apache.org/versioning-policy.html

Supporting new Java version is considered as a new feature which we cannot allow to backport.
Bug: allow configuring the max column name length in csv writer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Dec/20 07:48
Affects Version/s: 3.1.0

csv writer actually has an implicit limit on column name length due to univocity-parser, 

 

when we initialize a writer [https://github.com/uniVocity/univocity-parsers/blob/e09114c6879fa6c2c15e7365abc02cda3e193ff7/src/main/java/com/univocity/parsers/common/AbstractWriter.java#L211,] it calls toIdentifierGroupArray which calls valueOf in NormalizedString.java eventually ([https://github.com/uniVocity/univocity-parsers/blob/e09114c6879fa6c2c15e7365abc02cda3e193ff7/src/main/java/com/univocity/parsers/common/NormalizedString.java#L205-L209)]

 

in that stringCache.get, it has a maxStringLength cap [https://github.com/uniVocity/univocity-parsers/blob/e09114c6879fa6c2c15e7365abc02cda3e193ff7/src/main/java/com/univocity/parsers/common/StringCache.java#L104] which is 1024 by default

 

we do not expose this as configurable option, leading to NPE when we have a column name larger than 1024, 

 

```

[info]   Cause: java.lang.NullPointerException:

[info]   at com.univocity.parsers.common.AbstractWriter.submitRow(AbstractWriter.java:349)

[info]   at com.univocity.parsers.common.AbstractWriter.writeHeaders(AbstractWriter.java:444)

[info]   at com.univocity.parsers.common.AbstractWriter.writeHeaders(AbstractWriter.java:410)

[info]   at org.apache.spark.sql.catalyst.csv.UnivocityGenerator.writeHeaders(UnivocityGenerator.scala:87)

[info]   at org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter$.writeHeaders(CsvOutputWriter.scala:58)

[info]   at org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter.<init>(CsvOutputWriter.scala:44)

[info]   at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anon$1.newInstance(CSVFileFormat.scala:86)

[info]   at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:126)

[info]   at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:111)

[info]   at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:269)

[info]   at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:210)

```

 

it could be reproduced by a simple unit test

 

```

val row1 = Row("a")
val superLongHeader = (0 until 1025).map(_ => "c").mkString("")
val df = Seq(s"${row1.getString(0)}").toDF(superLongHeader)
df.repartition(1)
 .write
 .option("header", "true")
 .option("maxColumnNameLength", 1025)
 .csv(dataPath)

```
Bug: gapply fails execution with rbind error
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Dec/20 11:50
Affects Version/s: 3.0.0

Executing following code on databricks runtime 7.3 LTS ML errors out showing some rbind error whereas it is successfully executed without enabling Arrow in Spark session. Full error message attached.

 

```

library(dplyr)
library(SparkR)

SparkR::sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = "true"))

mtcars %>%
 SparkR::as.DataFrame() %>%

SparkR::gapply(x = .,
 cols = c("cyl", "vs"),
 
 func = function(key,
 data){
 
 dt <- data[,c("mpg", "qsec")]
 res <- apply(dt, 2, mean)
 df <- data.frame(firstGroupKey = key[1],
 secondGroupKey = key[2],
 mean_mpg = res[1],
 mean_cyl = res[2])
 return(df)
 
 }, 
 schema = structType(structField("cyl", "double"),
 structField("vs", "double"),
 structField("mpg_mean", "double"),
 structField("qsec_mean", "double"))
 ) %>%
 display()

```
Bug: Is Spark SQL able to auto update partition stats like hive by setting hive.stats.autogather=true
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Dec/20 09:31
Affects Version/s: 3.0.1

{{`spark.sql.statistics.size.autoUpdate.enabled` is only work for table stats update.}}{{But for partition stats,I can only update it with `ANALYZE TABLE tablename PARTITION(part) COMPUTE STATISTICS`.So is Spark SQL able to auto update partition stats like hive by setting hive.stats.autogather=true?}}
Bug: Allow Spark Structured Streaming report Kafka Lag through Burrow
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Dec/20 23:12
Affects Version/s: 3.0.1

Because structured streaming tracks Kafka offset consumption by itself, 
It is not possible to track total Kafka lag using Burrow similar to DStreams

We have used Stream hooks as mentioned [here|https://medium.com/@ronbarabash/how-to-measure-consumer-lag-in-spark-structured-streaming-6c3645e45a37] 

It would be great if Spark supports this feature out of the box.
Bug: Expose DataStreamReader.table and DataStreamWriter.toTable to PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/20 05:03
Affects Version/s: 3.2.0

From SPARK-32885 and SPARK-32896 we added two public APIs to enable read/write with table, but only in Scala side so only JVM languages could leverage them.

Given there're lots of PySpark users, it would be great to expose these public APIs to PySpark as well.
Bug: Refactor usage of Executor in ExecutorSuite to ensure proper cleanup
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/20 13:20
Affects Version/s: 3.0.1

Recently an issue was discovered that leaked Executors (which are not explicitly stopped after a test) can cause other tests to fail due to the JVM being killed after 10 min.

It is therefore crucial that tests always stop the Executor. In most tests this is already the case, but to make this pattern more explicit it would be good to have a helper function that ensures stopping the executor in a finally block.
Bug: Throw NoSuchPartitionsException from HiveExternalCatalog.dropPartitions()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/20 08:46
Affects Version/s: 2.4.7

HiveExternalCatalog.dropPartitions and as a consequence of that V1 Hive catalog throws AnalysisException. The behavior deviates from V1/V2 in-memory catalogs that throw NoSuchPartitionsException.
Bug: Recover PySpark coverage in spark-master-test-sbt-hadoop-3.2 Jenkins job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/20 07:08
Affects Version/s: 3.2.0

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-3.2/1726/console is being failed as below:

{code}
------------------------------------------------------------------------------------
TOTAL                                              31290   4915   8002    769    82%
Generating HTML files for PySpark coverage under /home/jenkins/workspace/spark-master-test-sbt-hadoop-3.2/python/test_coverage/htmlcov
/home/jenkins/workspace/spark-master-test-sbt-hadoop-3.2
Cloning into 'pyspark-coverage-site'...

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.
{code}
Bug: Upgrade Kafka to 2.8.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Dec/20 15:49
Affects Version/s: 3.2.0

This issue aims to upgrade Kafka client to 2.8.0.
Note that Kafka 2.8.0 uses ZSTD JNI 1.4.9-1 like Apache Spark 3.2.0.

*RELEASE NOTE*
- https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html
- https://downloads.apache.org/kafka/2.7.0/RELEASE_NOTES.html

This will bring the latest client-side improvement and bug fixes like the following examples.

- KAFKA-10631 ProducerFencedException is not Handled on Offest Commit
- KAFKA-10134 High CPU issue during rebalance in Kafka consumer after upgrading to 2.5
- KAFKA-12193 Re-resolve IPs when a client is disconnected
- KAFKA-10090 Misleading warnings: The configuration was supplied but isn't a known config
- KAFKA-9263 The new hw is added to incorrect log when  ReplicaAlterLogDirsThread is replacing log 
- KAFKA-10607 Ensure the error counts contains the NONE
- KAFKA-10458 Need a way to update quota for TokenBucket registered with Sensor
- KAFKA-10503 MockProducer doesn't throw ClassCastException when no partition for topic
Bug: Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/20 02:44
Affects Version/s: 3.1.0

See [https://github.com/apache/spark/pull/21092#discussion_r540240095.]
We should respect PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON like we do in all other places.
Bug: improve the next-day function of the sql component to deal with Column type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/20 13:11
Affects Version/s: 3.0.0

Hello all,

 

I used the function next_day in the spark SQL component and loved it: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3077]

 

Actually the signature of this function is: def next_day(date: Column, dayOfWeek: String): Column.

It accepts the dayOfWeek parameter as a String. However in my case, the dayOfWeek is in a Column, so different values for each row of the dataframe. So I had to use the NextDay function like this: NextDay(dateCol.expr, dayOfWeekCol.expr).

 

My proposition is to add another signature for this function: def next_day(date: Column, dayOfWeek: Column): Column

 

In fact it is already the case for some other functions in this scala object, exemple:

def date_sub(start: Column, days: Int): Column = date_sub(start, lit(days))

def date_sub(start: Column, days: Column): Column = withExpr \{ DateSub(start.expr, days.expr) }

 

or 

 

def add_months(startDate: Column, numMonths: Int): Column = add_months(startDate, lit(numMonths))

def add_months(startDate: Column, numMonths: Column): Column = withExpr {
 AddMonths(startDate.expr, numMonths.expr)
 }

 

I hope have explained my idea clearly. Let me know what are your opinions. If you are ok, I can submit a pull request with the necessary change.

 

Kind regardes,

Chongguang
Bug: Test failures: ALTER TABLE .. DROP PARTITION tries to delete files out of partition path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Dec/20 15:17
Affects Version/s: 3.1.0

For example: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132719/testReport/org.apache.spark.sql.hive.execution.command/AlterTableAddPartitionSuite/ALTER_TABLE____ADD_PARTITION_Hive_V1__SPARK_33521__universal_type_conversions_of_partition_values/

{code:java}
org.apache.spark.sql.hive.execution.command.AlterTableAddPartitionSuite.ALTER TABLE .. ADD PARTITION Hive V1: SPARK-33521: universal type conversions of partition values
sbt.ForkMain$ForkError: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: File file:/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-38fe2706-33e5-469a-ba3a-682391e02179 does not exist;
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:112)
	at org.apache.spark.sql.hive.HiveExternalCatalog.dropPartitions(HiveExternalCatalog.scala:1014)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.dropPartitions(ExternalCatalogWithListener.scala:211)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.dropPartitions(SessionCatalog.scala:1036)
	at org.apache.spark.sql.execution.command.AlterTableDropPartitionCommand.run(ddl.scala:582)
{code}
Bug: Spark Core ::Spark core versions upto 3.0.1 using interdependency on Jackson-core-asl version 1.9.13, which is having security issues reported. 
Status: Reopened | Priority: Major | Resolution: 
Created: 10/Dec/20 09:56
Affects Version/s: 3.0.1

spark-core version upto latest 3.0.1 is using dependency [org.apache.avro|https://mvnrepository.com/artifact/org.apache.avro] version 1.8.2 which is having [jackson-core-asl|https://mvnrepository.com/artifact/org.codehaus.jackson/jackson-core-asl] version 1.9.13 which has security issues.
Please fix and share the new version.
Bug: ANSI mode: Casting string to date should throw exception on parse error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/20 16:51
Affects Version/s: 3.2.0

When casting a string as timestamp in ANSI mode, Spark throws a runtime exception on parsing error.
However, the result for casting a string to date is always null.  We should throw an exception on parsing error as well.
Bug: `gpg: keyserver receive failed: No name` during K8s IT
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/20 23:02
Affects Version/s: 3.0.2

K8s IT fails with gpg: keyserver receive failed: No name. This seems to be consistent in the new Jenkins Server.

{code}
Executing: /tmp/apt-key-gpghome.gGqC9RwptN/gpg.1.sh --keyserver keys.gnupg.net --recv-key E19F5F87128899B192B1A2C2AD5F960A256A04AF
gpg: keyserver receive failed: No name
The command '/bin/sh -c echo "deb http://cloud.r-project.org/bin/linux/debian buster-cran35/" >> /etc/apt/sources.list &&   apt install -y gnupg &&   apt-key adv --keyserver keys.gnupg.net --recv-key 'E19F5F87128899B192B1A2C2AD5F960A256A04AF' &&   apt-get update &&   apt install -y -t buster-cran35 r-base r-base-dev &&   rm -rf /var/cache/apt/*' returned a non-zero code: 2
{code}

It locally works on Mac.
{code}
$ gpg1 --keyserver keys.gnupg.net --recv-key E19F5F87128899B192B1A2C2AD5F960A256A04AF
gpg: requesting key 256A04AF from hkp server keys.gnupg.net
gpg: key 256A04AF: public key "Johannes Ranke (Wissenschaftlicher Berater) <johannes.ranke@jrwb.de>" imported
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   2  signed:   1  trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1  valid:   1  signed:   0  trust: 1-, 0q, 0n, 0m, 0f, 0u
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
{code}


It happens multiple times.
- https://github.com/apache/spark/pull/30693
- https://github.com/apache/spark/pull/30694
Bug: Thrift server : FETCH_PRIOR does not cause to reiterate from start position. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/20 01:53
Affects Version/s: 3.1.0

Currently, when a client requests FETCH_PRIOR to thrift server, thrift server reiterates from start position. Because thrift server caches a query result with an array, FETCH_PRIOR can be implemented without reiterating the result.
Bug: Support multiple types of function partition pruning on hive metastore
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Dec/20 08:35
Affects Version/s: 3.2.0

For the current version, partition pruning support is limited to the scene.

Let's look at the implementation of the source code:
 [https://github.com/apache/spark/blob/031c5ef280e0cba8c4718a6457a44b6cccb17f46/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala#L840]

Hive getPartitionsByFilter() takes a string that represents partition predicates like "str_key=\"value\" and int_key=1 ...", but for normal functions like concat/concat_ws/substr,it  does not support.



The defect can cause a large number of partitions to be scanned which will increase the amount of data involved in the calculation and increase the pressure of service of metastore.
Bug: Broadcast timeout happened unexpectedly in AQE 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/20 08:23
Affects Version/s: 3.0.0

In Spark 3.0, when AQE is enabled, there is often broadcast timeout in normal queries as below.

 
{code:java}
Could not execute broadcast in 300 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1
{code}
 

This is usually happens when broadcast join(with or without hint) after a long running shuffle (more than 5 minutes).  By disable AQE, the issues disappear.

The workaround is to increase spark.sql.broadcastTimeout and it works. But because the data to broadcast is very small, that doesn't make sense.

After investigation, the root cause should be like this: when enable AQE, in getFinalPhysicalPlan, spark traversal the physical plan bottom up and create query stage for materialized part by createQueryStages and materialize those new created query stages to submit map stages or broadcasting. When ShuffleQueryStage are materializing before BroadcastQueryStage, the map job and broadcast job are submitted almost at the same time, but map job will hold all the computing resources. If the map job runs slow (when lots of data needs to process and the resource is limited), the broadcast job cannot be started(and finished) before spark.sql.broadcastTimeout, thus cause whole job failed (introduced in SPARK-31475).

Code to reproduce:

 
{code:java}
import java.util.UUID
import scala.util.Random
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .master("local[2]")
  .appName("Test Broadcast").getOrCreate()
import spark.implicits._

spark.conf.set("spark.sql.adaptive.enabled", "true")

val sc = spark.sparkContext
sc.setLogLevel("INFO")
val uuid = UUID.randomUUID

val df = sc.parallelize(Range(0, 10000), 10000).flatMap(x => {
  for (i <- Range(0, 10000 + Random.nextInt(10000)))
    yield (x % 26, x, Random.nextInt(10), UUID.randomUUID.toString)
}).toDF("index", "part", "pv", "uuid")
  .withColumn("md5", md5($"uuid"))

val dim_data = Range(0, 26).map(x => (('a' + x).toChar.toString, x))
val dim = dim_data.toDF("name", "index")

val result = df.groupBy("index")
  .agg(sum($"pv").alias("pv"), countDistinct("uuid").alias("uv"))
  .join(dim, Seq("index"))
  .collect(){code}