av9ash/GitBugs huggingface dataset:
Bug: Spark application submitted despite failing to get Hive delegation token
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/21 02:50
Affects Version/s: 3.1.1

Currently, when running on YARN and failing to get Hive delegation token, a Spark SQL application will still be submitted. Eventually, the application will fail on connecting to Hive metastore without a valid delegation token. 

Is there any reason for this design ?

cc [~jerryshao] who originally implemented this in https://issues.apache.org/jira/browse/SPARK-14743

I'd propose to fail immediately like HadoopFSDelegationTokenProvider.

 

Update:

After [https://github.com/apache/spark/pull/23418], HadoopFSDelegationTokenProvider no longer fail on non fatal exception. However, the author changed the behavior just to keep it consistent with other providers.
Bug: Recover the interval case in the benchmark of ExtractBenchmark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/21 01:52
Affects Version/s: 3.2.0

{code}
Running benchmark: Invoke extract for interval
  Running case: cast to interval
21/04/02 10:40:01 INFO BlockManagerInfo: Removed broadcast_349_piece0 on 192.168.35.219:55076 in memory (size: 7.3 KiB, free: 434.4 MiB)
21/04/02 10:40:01 INFO BlockManagerInfo: Removed broadcast_348_piece0 on 192.168.35.219:55076 in memory (size: 7.3 KiB, free: 434.4 MiB)
21/04/02 10:40:01 INFO BlockManagerInfo: Removed broadcast_345_piece0 on 192.168.35.219:55076 in memory (size: 7.3 KiB, free: 434.4 MiB)
21/04/02 10:40:01 INFO BlockManagerInfo: Removed broadcast_346_piece0 on 192.168.35.219:55076 in memory (size: 7.3 KiB, free: 434.4 MiB)
21/04/02 10:40:01 INFO BlockManagerInfo: Removed broadcast_347_piece0 on 192.168.35.219:55076 in memory (size: 7.3 KiB, free: 434.4 MiB)
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'subtractdates(CAST(timestamp_seconds(id) AS DATE), DATE '0001-01-01') + subtracttimestamps(timestamp_seconds(id), TIMESTAMP '1000-01-01 01:02:03.123456')' due to data type mismatch: argument 1 requires timestamp type, however, 'subtractdates(CAST(timestamp_seconds(id) AS DATE), DATE '0001-01-01')' is of day-time interval type.; line 1 pos 0;
'Project [unresolvedalias(cast(subtractdates(cast(timestamp_seconds(id#1400L) as date), 0001-01-01, false) + subtracttimestamps(timestamp_seconds(id#1400L), 1000-01-01 01:02:03.123456) as day-time interval), Some(org.apache.spark.sql.Column$$Lambda$1282/0x0000000800bd5040@5bff7f13))]
+- Range (1262304000, 1272304000, step=1, splits=Some(1))
{code}

Benchmark case is broken.
Bug: Support columns batch adding in PySpark.dataframe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/21 09:30
Affects Version/s: 3.2.0

Now, the pyspark can only use withColumn to do column adding a column or replacing the existing column that has the same name. The scala withColumn can adding columns at one pass. [1]

 

Before this added, the user can only use withColumn again and again like:

 
{code:java}
self.df.withColumn("key1", col("key1")).withColumn("key2", col("key2")).withColumn("key3", col("key3")){code}
 

After the support, you user can use the with_columns complete batch operations:

 
{code:java}
self.df.withColumn(["key1", "key2", "key3"], [col("key1"), col("key2"), col("key3")]){code}
 

[1] [https://github.com/apache/spark/blob/b5241c97b17a1139a4ff719bfce7f68aef094d95/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2402]
Bug: Establish error message guidelines
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/21 21:15
Affects Version/s: 3.1.0

In the SPIP: Standardize Exception Messages in Spark, there are three major improvements proposed:

# Group error messages in dedicated files.
# Establish an error message guideline for developers.
# Improve error message quality.

The second step is to establish the error message guideline. This was discussed in http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Build-error-message-guideline-td31076.html and added to the website in https://github.com/apache/spark-website/pull/332. To increase visibility, the guidelines should be accessible from the PR template.
Bug: Declare the markdown package as a dependency of the SparkR package
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/21 08:54
Affects Version/s: 3.1.0

If we didn't install pandoc locally, the make-distribution package will fail with the following message:
{quote}
--- re-building ‘sparkr-vignettes.Rmd’ using rmarkdown
Warning in engine$weave(file, quiet = quiet, encoding = enc) :
  Pandoc (>= 1.12.3) not available. Falling back to R Markdown v1.
Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:
The 'markdown' package should be declared as a dependency of the 'SparkR' package (e.g., in the  'Suggests' field of DESCRIPTION), because the latter contains vignette(s) built with the 'markdown' package. Please see https://github.com/yihui/knitr/issues/1864 for more information.
--- failed re-building ‘sparkr-vignettes.Rmd’
{quote}
Bug: org.apache.spark.sql.AnalysisException: Resolved attribute(s) colName#390 missing from listofcolumns
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 12/Apr/21 20:39
Affects Version/s: 3.1.1

I create a column to exchange an existing one in my dataframe using
{code:java}
withColumn {code}
like so : 
{noformat}
Dataset<Row> df2 = spark.createDataset(data, Encoders.DOUBLE()).toDF("colName1");
datasetRow.withColumn("colName1",df2.col("colName1"));{noformat}
But the last line of code is yielding the following : 

org.apache.spark.sql.AnalysisException: Resolved attribute(s) colName1#390 missing from ...

Why ?
Bug: Wrong memory allocation on standalone mode cluster
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/Apr/21 07:34
Affects Version/s: 3.0.1

I see a bug in executer memory allocation in the standalone cluster, but I can't find which part of the spark code causes this problem. That why's I decided to raise this issue here.

Assume you have 3 workers with 10 CPU cores and 10 Gigabyte memories. Assume also you have 2 spark jobs that run on this cluster of workers, and these jobs configs set as below:

-----------------

job-1:

executer-memory: 5g

executer-CPU: 4

max-cores: 8

------------------

job-2:

executer-memory: 6g

executer-CPU: 4

max-cores: 8

------------------

In this situation, We expect that if we submit both of these jobs, the first job that submits get  2 executers which each of them has 4 CPU core and 5g memory, and the second job gets only one executer on thirds worker who has 4 CPU core and 6g memory because worker 1 and worker 2 doesn't have enough memory to accept the second job. But surprisingly, we see that one of the first or second workers creates an executor for job-2, and the worker's consuming memory goes beyond what's allocated to that and gets 11g memory from the operating system.

Is this behavior normal? I think this can cause some undefined behavior problem in the cluster.
Bug: abs return negative
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Apr/21 02:50
Affects Version/s: 3.0.1

from pyspark.sql.functions import lit, abs, concat, hash,col

df = df.withColumn('partition_id', lit(-2147483648))
 df = df.withColumn('abs_id', abs(col('partition_id')))
 df.select('abs_id','partition_id').show()

 

when the number is  -2147483648，method abs return negative 

 +-----------+------------+
| abs_id        |partition_id |
+-----------+------------+
|-2147483648| -2147483648|
|-2147483648| -2147483648|
|-2147483648| -2147483648|
|-2147483648| -2147483648|
|-2147483648| -2147483648|
+-----------+------------+
Bug: Add Data Source Option links to missing documents.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 01:39
Affects Version/s: 3.2.0

We're missing the documents for options in detail such as `to_xxx`, `from_xxx`, `schema_of_csv` and `schema_of_json` functions.

We should add a link to the Data Source Option for them.
Bug: Port/integrate Koalas remaining codes into PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/21 20:10
Affects Version/s: 3.2.0

There are some more commits remaining after the main codes were ported.

- [https://github.com/databricks/koalas/commit/c8f803d6becb3accd767afdb3774c8656d0d0b47]
- [https://github.com/databricks/koalas/commit/913d68868d38ee7158c640aceb837484f417267e]
Bug: SQL DataFrameReader unescapedQuoteHandling parameter is misdocumented
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/21 01:39
Affects Version/s: 3.1.2

The unescapedQuoteHandling parameter of DataFrameReader isn't correctly documented. STOP_AT_DELIMITER appears twice, and it looks like that's overwritten an intended option, e.g. [https://github.com/apache/spark/blob/1d550c4e90275ab418b9161925049239227f3dc9/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L744-L749]

To view instances where this error occurs, this is a useful query [https://github.com/apache/spark/search?q=STOP_AT_DELIMITER]

It appears that this bug was introduced here: [https://github.com/apache/spark/pull/30518|https://github.com/apache/spark/pull/30518,]
Bug: org.apache.spark.SparkException: Task not serializable
Status: Open | Priority: Major | Resolution: 
Created: 29/Apr/21 12:53
Affects Version/s: 3.0.1

I am getting NotSerializableException when broadcasting Serialized class.

 

Minimal code with which you can reproduce this issue

 
{code:java}
case class Student(name: String)

class NonSerializable() {

  def getText() : String = {
    """
      |[
      |{
      | "name": "test1"
      |},
      |{
      | "name": "test2"
      |}
      |]
      |""".stripMargin
  }
}

val obj = new NonSerializable()
val descriptors_string = obj.getText()


import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._

val parsed_descriptors: Array[Student] =
  readFromString[Array[Student]](descriptors_string)(JsonCodecMaker.make)

val broadcast_descriptors = spark.sparkContext.broadcast(parsed_descriptors)

def foo(data: String): Seq[Any] = {

  import scala.collection.mutable.ArrayBuffer
  val res = new ArrayBuffer[String]()
  for (desc <- broadcast_descriptors.value) {
    res += desc.name
  }
  res
}

val data = spark.sparkContext.parallelize(Array("1", "2", "3")).map(x => foo(x)).collect()
{code}
Command used to start spark-shell
{code:java}
./bin/spark-shell --master local --jars /Users/sandeep.katta/.m2/repository/com/github/plokhotnyuk/jsoniter-scala/jsoniter-scala-macros_2.12/2.7.1/jsoniter-scala-macros_2.12-2.7.1.jar,/Users/sandeep.katta/.m2/repository/com/github/plokhotnyuk/jsoniter-scala/jsoniter-scala-core_2.12/2.7.1/jsoniter-scala-core_2.12-2.7.1.jar
{code}
*Exception Details*
{code:java}
org.apache.spark.SparkException: Task not serializable
  at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:416)
  at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:406)
  at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
  at org.apache.spark.SparkContext.clean(SparkContext.scala:2362)
  at org.apache.spark.rdd.RDD.$anonfun$map$1(RDD.scala:396)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
  at org.apache.spark.rdd.RDD.map(RDD.scala:395)
  ... 51 elided
Caused by: java.io.NotSerializableException: NonSerializable
Serialization stack:
	- object not serializable (class: NonSerializable, value: NonSerializable@7c95440d)
	- field (class: $iw, name: obj, type: class NonSerializable)
	- object (class $iw, $iw@3ed476a8)
	- field (class: $iw, name: $iw, type: class $iw)
	- object (class $iw, $iw@381f7b8e)
	- field (class: $iw, name: $iw, type: class $iw)
	- object (class $iw, $iw@23635e39)
	- field (class: $iw, name: $iw, type: class $iw)
	- object (class $iw, $iw@1a8b3791)
	- field (class: $iw, name: $iw, type: class $iw)
{code}
Bug: In k8s, during running spark job, IllegalArgumentException(too large frame) is raised on spark driver. It seems to related to prometheus
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Apr/21 06:50
Affects Version/s: 3.0.2

In k8s, during running spark job, IllegalArgumentException(too large frame) is raised on spark driver like that
{code:java}
2021-04-26 13:59:26,961 WARN server.TransportChannelHandler: Exception in connection from /${Prometheus pod Ip addr}:51346
java.lang.IllegalArgumentException: Too large frame: 5135603447297303916
at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119)
at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source) {code}
 

${Prometheus pod Ip addr} means prometheus pod's cluster ip in k8s.

How to resolve the warning?
Bug: Improve LiveEntityHelpers.newAccumulatorInfos performace
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/21 01:58
Affects Version/s: 3.2.0

[It|https://github.com/apache/spark/blob/3854ad87c78f2a331f9c9c1a34f9ec281900f8fe/core/src/main/scala/org/apache/spark/status/LiveEntity.scala#L646-L661] will impact performance if  there are lot of {{AccumulableInfo}} instances.

{noformat}
 num     #instances         #bytes  class name
----------------------------------------------
   1:     331897850    50708371592  [C
   2:        795358    24591381896  [J
   3:     330836234    10586759488  java.lang.String
   4:     139726297     8942483008  org.apache.spark.sql.execution.metric.SQLMetric
   5:     249040156     5976963744  scala.Some
   6:     146278719     5851148760  org.apache.spark.util.AccumulatorMetadata
   7:      38448113     5536528272  java.net.URI
   8:      37540162     3603855552  org.apache.hadoop.fs.FileStatus
   9:      69724130     3346758240  java.util.Hashtable$Entry
  10:      61521559     2953034832  java.util.concurrent.ConcurrentHashMap$Node
  11:      50421974     2823630544  scala.collection.mutable.LinkedEntry
  12:      43349222     2774350208  org.apache.spark.scheduler.AccumulableInfo
{noformat}


{noformat}
--- 15430388364 ns (2.03%), 1543 samples
  [ 0] scala.collection.TraversableLike.noneIn$1
  [ 1] scala.collection.TraversableLike.filterImpl
  [ 2] scala.collection.TraversableLike.filterImpl$
  [ 3] scala.collection.AbstractTraversable.filterImpl
  [ 4] scala.collection.TraversableLike.filter
  [ 5] scala.collection.TraversableLike.filter$
  [ 6] scala.collection.AbstractTraversable.filter
  [ 7] org.apache.spark.status.LiveEntityHelpers$.newAccumulatorInfos
  [ 8] org.apache.spark.status.LiveTask.doUpdate
  [ 9] org.apache.spark.status.LiveEntity.write
{noformat}
Bug: nullable field is set to false for integer type when using reflection to get StructType for a case class
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Apr/21 05:32
Affects Version/s: 3.1.1

{code:java}
// code placeholder

object Util {
  def toStructType[T](implicit typeTags: ScalaReflection.universe.TypeTag[T]): StructType =
  ScalaReflection.schemaFor[T].dataType.asInstanceOf[StructType]
}

case class MyTable(val a: Int, val b: String)

// The following test fails because schema returned using reflection sets nullable=false for integer column. The test passes if you set it to false

it must "return a Spark Schema of type StructType for a case class" in {
 val schemaFromCaseClass: StructType = Util.toStructType[MyTable]
 val expectedSchema = new StructType().add(StructField("a", IntegerType, true)).add(StructField("b", StringType, true))
 schemaFromCaseClass.size mustBe 2
 schemaFromCaseClass mustBe expectedSchema
}




{code}
Bug: Use permissive and squashed merge when syncing to the latest branch in GitHub Actions testing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 03:00
Affects Version/s: 3.2.0

Please see https://github.com/apache/spark/pull/32168

There is an issue when syncing to the Apache master branch.

{code}
From https://github.com/yaooqinn/spark
 * branch                  SPARK-35044 -> FETCH_HEAD
fatal: Not possible to fast-forward, aborting.
Error: Process completed with exit code 128.
{code}

This is because we use {{--ff-only}} option so it assumes that the fork is always based on the latest master branch.

We should make it less strict
Bug: Deprecate Apache Mesos as resource manager
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/21 13:58
Affects Version/s: 3.2.0

As highlighted in https://lists.apache.org/thread.html/rab2a820507f7c846e54a847398ab20f47698ec5bce0c8e182bfe51ba%40%3Cdev.mesos.apache.org%3E , Apache Mesos is moving to the attic and ceasing development.

We can/should maintain support for some time, but, can probably go ahead and deprecate it now.
Bug: Automated formatting for Scala Code for Blank Lines.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/21 12:59
Affects Version/s: 3.2.0

Based on the databricks' scala-style-code for blanklines.  [https://github.com/databricks/scala-style-guide#blanklines]

Add a configuration to controls whether to enforce a blank line before and/or after a top-level statement spanning a certain number of lines.

And upgrade mvn-scalafmt plugin to 1.0.4 to enable this configuration.
Bug: Aggregate ordinal should judge is it contains agg function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/21 06:02
Affects Version/s: 3.2.0

Aggregate ordinal should judge is it contains agg function.

before this change it's 
```

-- !query
select a, b, sum(b) from data group by 3
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.AnalysisException
aggregate functions are not allowed in GROUP BY, but found sum(data.b)

```
Bug: Transfer ANSI intervals via Hive Thrift server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/21 09:24
Affects Version/s: 3.2.0

Currently, Hive Thrift server cannot recognize new ANSI interval types:

{code:java}
 $ ./sbin/start-thriftserver.sh
 $ ./bin/beeline
Beeline version 2.3.8 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/default "" "" ""
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Spark SQL (version 3.2.0-SNAPSHOT)
0: jdbc:hive2://localhost:10000/default> select timestamp'2021-01-01 01:02:03.000001' - date'2020-12-31';
Error: java.lang.IllegalArgumentException: Unrecognized type name: day-time interval (state=,code=0)
{code}
Bug: Remove deprecated syntex in SBT build file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/21 10:35
Affects Version/s: 3.2.0

Related to SPARK-34959

SBT 1.5.0 deprecates {{in}} syntax from 0.13.x, so build file adjustment
 is recommended.
 See [https://www.scala-sbt.org/1.x/docs/Migrating-from-sbt-013x.html#Migrating+to+slash+syntax]
Bug: from_json() acts differently on created and literal strings with backslashes
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 08/Apr/21 17:36
Affects Version/s: 3.1.1

JSON string with the value that contains backslashes fails to be recovered by `from_json()`.

I found that if the same string is created with `to_json(named_struct())` then it actually does work.

 

The following code to reproduce. I would expect all of these methods to return the same (correct) result:
{code:java}
select to_json(named_struct('msg', '\\'))
 , schema_of_json(to_json(named_struct('msg', '\\')))
 , from_json(to_json(named_struct('msg', '\\')), schema_of_json(to_json(named_struct('msg', '\\'))))
 , from_json('{"msg":"\\"}', schema_of_json(to_json(named_struct('msg', '\\'))))
 , from_json('{"msg":"\\"}', 'msg string')
 
{code}
 

!image-2021-04-11-07-21-02-750.png!
Bug: Index Appending when Duplicate Column names
Status: Open | Priority: Major | Resolution: 
Created: 08/Apr/21 15:21
Affects Version/s: 3.1.1

In reference to below mentioned link
[https://github.com/apache/spark/pull/14745]
I am reporting this issue that can it should be configurable to append values when we have duplicate columns? For instance, instead of always appending column index, there should be configurable leverage to append any sort of string. Lets say, instead of Cola2 and Cola34, we can set as Cola[xyz] and Cola1.
Bug: Spark hangs after on doing multiple unions
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Apr/21 13:19
Affects Version/s: 3.1.1

I'm trying to union a list of {{Dataset<Rows>}} like so:

 

{{}}
{code:java}
totalSet = sparkSession.emptyDataset(RowEncoder.apply(schema)); 
for(var currentDataset : datasetList)
     totalSet = totalSet.union(currentDataset);
totalSet.show(10000,false);
{code}
{{}}

There's about 160 datasets in datasetList. totalSet should only have 1k rows at the end of the loop. However, spark gets stuck here in this loop. If I union just 1 dataset from datasetList it works just fine. I thought doing unions ought to be a O(1) operation, but that doesn't seem to be the case. Does anyone know why this code hangs?
Bug: Support time-travel SQL syntax
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 07/Apr/21 06:40
Affects Version/s: 3.1.1

Some DataSource have the ability to query the older snapshot, like Delta.

The syntax may be like `TIMESTAMP AS OF '

2018-10-18 22:15:12' ` to query the snapshot closest to this time point, or `VERSION AS OF 12` to query the 12th snapshot.
Bug: pyspark 2.4.7 in conda
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Apr/21 18:03
Affects Version/s: 2.4.7

Pyspark 2.4.7 does not seem to be available yet in conda: [https://anaconda.org/conda-forge/pyspark/files] shows only versions up to 2.4.6 (for major version 2).
{noformat}
conda install pyspark=2.4.7{noformat}
gives the following:

 

 
{code:java}
ResolvePackageNotFound:
13:44:43
    - pyspark=2.4.7
{code}
It would be great if this package were available through conda!