av9ash/GitBugs huggingface dataset:
Bug: cloudpickle: ValueError: Cell is empty
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/21 07:26
Affects Version/s: 3.1.2

{code:java}
  File "/Users/tekumara/code/awesome-spark-app/.venv/lib/python3.7/site-packages/pyspark/serializers.py", line 437, in dumps
    return cloudpickle.dumps(obj, pickle_protocol)
  File "/Users/tekumara/code/awesome-spark-app/.venv/lib/python3.7/site-packages/pyspark/cloudpickle/cloudpickle_fast.py", line 101, in dumps
    cp.dump(obj)
  File "/Users/tekumara/code/awesome-spark-app/.venv/lib/python3.7/site-packages/pyspark/cloudpickle/cloudpickle_fast.py", line 540, in dump
    return Pickler.dump(self, obj)
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/tekumara/code/awesome-spark-app/.venv/lib/python3.7/site-packages/pyspark/cloudpickle/cloudpickle_fast.py", line 722, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/Users/tekumara/code/awesome-spark-app/.venv/lib/python3.7/site-packages/pyspark/cloudpickle/cloudpickle_fast.py", line 659, in _save_reduce_pickle5
    dictitems=dictitems, obj=obj
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/Users/tekumara/.pyenv/versions/3.7.9/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/tekumara/code/awesome-spark-app/.venv/lib/python3.7/site-packages/dill/_dill.py", line 1226, in save_cell
    f = obj.cell_contents
ValueError: Cell is empty
{code}
Doesn't occur in Spark 3.0.0, so possibly introduced when cloudpickle was upgraded to 1.5.0 (see https://issues.apache.org/jira/browse/SPARK-32094).

Also doesn't occur in Spark 3.1.2 with python 3.8.
Bug: Using openGauss ( like postgres) as Spark Datasource, Spark thorws cast errors
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 14/Aug/21 14:47
Affects Version/s: 3.3.0

Hi, I'm trying to use openGauss as the Spark Datasource and implementing some datasource interfaces. I would like to know if there is any solution. Thank you!

 

But when I try to get the column which is VARCHAR type, I got an error like this:

!image-2021-08-14-22-47-56-474.png!

When I use getInt() or getDouble() for other type, the code  is working fine.

And here is some code:

 
{code:java}
object OpenGaussTable {
  /*Table products*/
  val schema: StructType = new StructType().add("name", StringType)

}
...

class OpenGaussPartitionReader(connectionProperties: ConnectionProperties) extends PartitionReader[InternalRow] {
  private val connection = DriverManager.getConnection(
    connectionProperties.url, connectionProperties.user, connectionProperties.password
  )
  private val statement = connection.createStatement()
  private val resultSet = statement.executeQuery(s"select * from ${connectionProperties.tableName}")

  override def next(): Boolean = resultSet.next()

  override def get(): InternalRow = InternalRow(resultSet.getString(1))

  override def close(): Unit = connection.close()
}{code}
 

the complete code address：

[https://pastebin.ubuntu.com/p/vCBssjhc2r/]

[https://pastebin.ubuntu.com/p/jCcDbb79NG/]
Bug: LSHModel.approxSimilarityJoin can generate invalid column names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/21 19:27
Affects Version/s: 3.1.2

"approxSimilarityJoin for self join" in BucketedRandomProjectionLSHSuite can fail with:
        org.apache.spark.sql.AnalysisException: cannot resolve '`keys#á¼ç°©`.`ì™ã‹`' given input columns: [keys#á¼ç°©.ì™ã‹];
Bug: The temp_shuffle files are not cleaned up when a task is interrupted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/21 18:22
Affects Version/s: 2.4.8

We discovered a potential file leak on the long running cluster. As the cluster running for a few hours, there are many "temp_shuffle_*" files left in the folder. It seems the files are not cleaned up as expected when the task finishes.
Bug: Add non-unified memory metrics
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Aug/21 02:49
Affects Version/s: 3.2.0

Spark split memory to managed memory (storage + execution) and un-managed memory. And use `spark.memory.fraction` to adjust the fraction of this to memory.
But we have no metrics to know how much we used for unmanaged memory and we have no data evidence to optimize configuration of `spark.memory.fraction`. Maybe we should add it
Bug: Creating Hive managed table from spark to Hive, results in the table being in the external directory
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/21 12:57
Affects Version/s: 2.4.5

Hi, 

Whenever I try creating a managed table in Hive from Spark,
 * the tables get created in external directory in HDFS with external type and format.
 * However, I tried dropping the table from Hive and it deleted the metadata as well as the actual data in the table.
Bug: Date Value '0001-01-01' changes to '0001-12-30' when inserted into a parquet hive table
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 09/Aug/21 16:00
Affects Version/s: 2.4.4

Hi All, 

we are seeing this issue on spark 2.4.4. Below are the steps to reproduce it. 

*Login in to hive terminal on cluster and create below tables.*

create table t_src(dob timestamp);
insert into t_src values('0001-01-01 00:00:00.0');
create table t_tgt(dob timestamp) stored as parquet;

 

*Spark-shell steps :*
 
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

val q0 = "TRUNCATE table t_tgt"
val q1 = "SELECT alias.dob as a0 FROM t_src alias"
val q2 = "INSERT INTO TABLE t_tgt SELECT tbl0.a0 as c0 FROM tbl0"
sqlContext.sql(q0)
sqlContext.sql(q1).select("a0").createOrReplaceTempView("tbl0")
sqlContext.sql(q2)

 

 After this check the contents of target table t_tgt. You will see the date "0001-01-01 00:00:00" changed to "0001-12-30 00:00:00".

 select * from t_tgt;

Is this a known issue? Is it fixed in any subsequent releases?

Thanks & regards,

Sindhura Alluri
Bug: Fix documentation in spark-env.sh.template
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/21 03:22
Affects Version/s: 3.1.2

Some options in the "Options read in YARN client/cluster mode" section in spark-env.sh.template is read by other modes too (e.g. SPARK_CONF_DIR, SPARK_EXECUTOR_CORES, etc.), so we should re-document it to help users distinguish what's only read by YARN mode from what's not.
Bug: Downloading lintr dependencies fail on GA
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/21 01:29
Affects Version/s: 3.3.0

Downloading lintr dependencies on GA fails.
I re-triggered the GA job but it still fail with the same error.

{code}
 * installing *source* package ‘devtools’ ...
** package ‘devtools’ successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (devtools)

The downloaded source packages are in
	‘/tmp/Rtmpv53Ix4/downloaded_packages’
Using bundled GitHub PAT. Please add your own PAT to the env var `GITHUB_PAT`
Error: Failed to install 'unknown package' from GitHub:
  HTTP error 401.
  Bad credentials

  Rate limit remaining: 59/60
  Rate limit reset at: 2021-08-06 01:37:46 UTC

  
Execution halted
Error: Process completed with exit code 1.
{code}

https://github.com/apache/spark/runs/3257853825
Bug: Unrecognized connection property 'url' when using Presto JDBC
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Aug/21 18:12
Affects Version/s: 3.1.1

Hi,  Here is my spark sql code, where I am trying to read a presto table based on this guide;  [https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html]
{code:scala}
 val df = spark.read
 .format("jdbc")
 .option("driver", "com.facebook.presto.jdbc.PrestoDriver")
 .option("url", "jdbc:presto://localhost:8889/mycatalog")
 .option("query", "select * from mydb.mytable limit 1")
 .option("user", "myuserid")
 .load()

{code}
 
 I am getting the following exception: *_unrecognized connection property 'url'_* 
{code:java}
Exception in thread "main" java.sql.SQLException: Unrecognized connection property 'url'
 at com.facebook.presto.jdbc.PrestoDriverUri.validateConnectionProperties(PrestoDriverUri.java:345)
 at com.facebook.presto.jdbc.PrestoDriverUri.<init>(PrestoDriverUri.java:102)
 at com.facebook.presto.jdbc.PrestoDriverUri.<init>(PrestoDriverUri.java:92)
 at com.facebook.presto.jdbc.PrestoDriver.connect(PrestoDriver.java:87)
 at org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:49)
 at org.apache.spark.sql.execution.datasources.jdbc.connection.ConnectionProvider$.create(ConnectionProvider.scala:68)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.$anonfun$createConnectionFactory$1(JdbcUtils.scala:62)
 at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:56)
 at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
 at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
 at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
 at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
 at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226)
 at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:341)
 

{code}
Seems like this issue is related to [https://github.com/prestodb/presto/issues/9254] where the property `url` is not a recognized property in Presto and looks like the fix needs to be done on the Spark side?

Our development is blocked on this exception and would appreciate any guidance. Thanks!

PS:
 presto-jdbc version: 0.245 / 0.260
Bug: Inconsistencies in approxQuantile annotations
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 30/Aug/21 21:36
Affects Version/s: 3.1.0

I've been reviewing PR in the legacy repo (https://github.com/zero323/pyspark-stubs/pull/552) and it looks like we have two problems with annotations for {{approxQuantile}}.

First of all {{DataFrame.approxQuantile}} should overload definition to match input arguments ‒ if col is a sequence then result should be a list of lists:

{code:python}
    @overload
    def approxQuantile(
        self,
        col: str,
        probabilities: Union[List[float], Tuple[float]],
        relativeError: float
    ) -> List[float]: ...
    @overload
    def approxQuantile(
        self,
        col: Union[List[str], Tuple[str]],
        probabilities: Union[List[float], Tuple[float]],
        relativeError: float
    ) -> List[List[float]]: ...
{code}


Additionally {{DataFrameStatFunctions.approxQuantile}} should match whatever we have in {{DataFrame}}.
Bug: Removal of netty-3.9.9.Final.jar dependent code and use only netty-all-4.1.47.Final.jar in Spark 2.4.x
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 24/Aug/21 09:10
Affects Version/s: 2.4.7

Please remove netty-3.9.9.Final.jar from Spark 2.4.x because this jar was reported for CVE-2019-20444 and CVE-2019-20445, we don't have option to upgrade netty-x.final.jar
Bug: Creating a value in json which represents decimal 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 25/Aug/21 14:56
Affects Version/s: 3.1.2

Hi,

 

I have a json which have some decimal values . This json is read by Spark like the following : 

Dataset<Row> df = spark.read().format("json").option("multiline","true").load("/home/user/work/Parquet/output.json");

 

The schema generated by this (df.printschema) shows the decimal value as double and not decimal .  

My requirement is that it should read the value as decimal only. 

Now to do this, I want change the value in json (while creating the json) in such a way or format that spark reads it as decimal only .

And in this conversion method, I need to apply the precision and scale also

Please let me know how this can be done 

 

Thanks and Regards

Anuradha
Bug: Allow column pruning on distributed sequence index (pandas API on Spark)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/21 06:18
Affects Version/s: 3.2.0

https://issues.apache.org/jira/browse/SPARK-36338 implemented distributed sequence implementation on JVM side. However, it disables leveraging Spark SQL engine because it creates an RDD directly, and truncate the SQL plans.

We should move the logic into a proper SQL plan to leverage other optimizations such as column pruning.
Bug: Remove `ColumnVector.numNulls`
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 23/Aug/21 07:51
Affects Version/s: 3.2.0

Hi!
When I was implementing ColumnVector abstract class I started to check where `numNulls` is used in spark source code. I didn't find any places where it is used except tests

Is there any plans to use it in the future. If no then I suppose to remove `numNulls` method.
Bug: Unable to create executor due to unable to connect to external shuffle server
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 18/Aug/21 11:41
Affects Version/s: 3.1.2

I am upgrading Spark version from 2.4.5 to 3.1.2. I am trying to to start standalone spark cluster using {{start-shuffle-service.sh}} in sbin but it seems like the file is not present in 3.1.2 version. Can you help me in understanding if I am doing it right? How do I start external shuffle server?
Bug: sparkstreaming input rate exceed  spark.streaming.kafka.maxRatePerPartition
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 19/Aug/21 11:25
Affects Version/s: 2.4.5

 

i have a spark streaming application with kafka . 
 Here are the parameters:

kafka partition = 500

batch time = 60 

--conf spark.streaming.backpressure.enabled=true

--conf spark.streaming.kafka.maxRatePerPartition=2500

input size= 500 * 120 * 2500 = 75,000,000

however input size become 160000000 after some batch 

who can tell me reason

!kafka.png!
Bug: varchar datatype behave differently on  hive table  and datasource table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/21 09:47
Affects Version/s: 2.3.1

in spark 3.1.X, when set spark.sql.hive.convertMetastoreOrc=false,and 

spark.sql.legacy.charVarcharAsString=true.

Execute the following sql:

CREATE TABLE t (col varchar(2)) stored as orc;
INSERT INTO t SELECT 'aaa';
select * from t;

result is aa

 

But when set spark.sql.hive.convertMetastoreOrc=true,and 

spark.sql.legacy.charVarcharAsString=true

alse execute the sql, the result is "aaa"
Bug: DS V2 Index Support
Status: Open | Priority: Major | Resolution: 
Created: 16/Aug/21 17:17
Affects Version/s: 3.3.0

Many data sources support index to improvement query performance. In order to take advantage of the index support in data source, the following APIs will be added for working with indexes:

{code:java}
 public interface SupportsIndex extends Table {

  /**
   * Creates an index.
   *
   * @param indexName the name of the index to be created
   * @param indexType the type of the index to be created. If this is not specified, Spark
   *                  will use empty String.
   * @param columns the columns on which index to be created
   * @param columnsProperties the properties of the columns on which index to be created
   * @param properties the properties of the index to be created
   * @throws IndexAlreadyExistsException If the index already exists.
   */
  void createIndex(String indexName,
      String indexType,
      NamedReference[] columns,
      Map<NamedReference, Map<String, String>> columnsProperties,
      Map<String, String> properties)
      throws IndexAlreadyExistsException;

  /**
   * Drops the index with the given name.
   *
   * @param indexName the name of the index to be dropped.
   * @throws NoSuchIndexException If the index does not exist.
   */
  void dropIndex(String indexName) throws NoSuchIndexException;

  /**
   * Checks whether an index exists in this table.
   *
   * @param indexName the name of the index
   * @return true if the index exists, false otherwise
   */
  boolean indexExists(String indexName);

  /**
   * Lists all the indexes in this table.
   */
  TableIndex[] listIndexes();
}

{code}
Bug: Increase pandas API coverage in PySpark
Status: Open | Priority: Major | Resolution: 
Created: 03/Aug/21 20:45
Affects Version/s: 3.3.0

Increase pandas API coverage in PySpark.

 

Especially, pending PRs [https://github.com/databricks/koalas/pulls] should be ported. Existing tickets are created for porting purposes, please avoid working on that.
Bug: Use v2 commands by default
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Aug/21 14:16
Affects Version/s: 3.3.0

It's been a while after we introduce the v2 commands, and I think it's time to use v2 commands by default even for the session catalog, with a legacy config to fall back to the v1 commands.

We can do this one command by one command, with tests for both the v1 and v2 versions. The tests should help us understand the behavior difference between v1 and v2 commands, so that we can:
 # fix the v2 commands to match the v1 behavior
 # or accept the behavior difference and write migration guide

We can reuse the test framework built in https://issues.apache.org/jira/browse/SPARK-33381
Bug: pushDownPredicate=false should prevent push down filters to JDBC data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Aug/21 10:24
Affects Version/s: 3.2.0

Spark SQL includes a data source that can read data from other databases using JDBC. 
Spark also supports the case-insensitive option pushDownPredicate.
According to http://spark.apache.org/docs/latest/sql-data-sources-jdbc.html, If set pushDownPredicate to false, no filter will be pushed down to the JDBC data source and thus all filters will be handled by Spark.
But I find it still be pushed down to JDBC data source.
Bug: DS V2 Filter support
Status: Open | Priority: Major | Resolution: 
Created: 22/Aug/21 16:02
Affects Version/s: 3.3.0

The motivation of adding DSV2 filters:
The values in V1 filters are Scala types. When translating catalyst Expression to V1 filers, we have to call convertToScala to convert from Catalyst types used internally in rows to standard Scala types, and later convert Scala types back to Catalyst types. This is very inefficient. In V2 filters, we use Expression for filter values, so the conversion from Catalyst types to Scala types and Scala types back to Catalyst types are avoided.
Bug: Allow streaming queries with Trigger.Once run in multiple batches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/21 09:48
Affects Version/s: 3.2.0

Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data the queries can process is limited, or Spark driver will be out of memory. 

We should allow streaming queries with Trigger.Once run in multiple batches.
Bug: "Invalid usage of '*' in expression" error due to the feature of 'quotedRegexColumnNames' in some scenarios.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/21 06:45
Affects Version/s: 2.4.8

 In some cases, the error happens when the following property is set.
{code:java}
spark.sql("set spark.sql.parser.quotedRegexColumnNames=true")
{code}
*case 1:* 
{code:java}
spark-sql> create table tb_test as select 1 as col_a, 2 as col_b;
spark-sql> select `tb_test`.`col_a`  from tb_test;
1

spark-sql> set spark.sql.parser.quotedRegexColumnNames=true;

spark-sql> select `tb_test`.`col_a`  from tb_test;
Error in query: Invalid usage of '*' in expression 'unresolvedextractvalue'


{code}
 

*case 2:*
{code:java}
         > select `col_a`/`col_b` as `col_c` from (select 3 as `col_a` ,  3.14 as `col_b`);
0.955414

spark-sql> set spark.sql.parser.quotedRegexColumnNames=true;

spark-sql> select `col_a`/`col_b` as `col_c` from (select 3 as `col_a` ,  3.14 as `col_b`);

Error in query: Invalid usage of '*' in expression 'divide'
{code}
 

This problem exists in 3.X, 2.4.X and master versions. 

 

Related issue ： 

https://issues.apache.org/jira/browse/SPARK-12139

（As can be seen in the latest comments, some people have encountered the same problem）

 

Similar problems：

https://issues.apache.org/jira/browse/SPARK-28897
Bug: Remove OptimizeSubqueries from batch of PartitionPruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Aug/21 07:28
Affects Version/s: 3.0.3

To support this case:

{code:scala}
sql(
        """
          |SELECT date_id, product_id FROM fact_sk f
          |JOIN (select store_id + 3 as new_store_id from dim_store where country = 'US') s
          |ON f.store_id = s.new_store_id
        """.stripMargin)

{code}
Bug: Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/21 05:20
Affects Version/s: 2.0.2

The `size` method of `ChunkedByteBufferOutputStream` returns a `Long` value; however, the underlying `_size` variable is initialized as `Int`.

That causes an overflow and returns a negative size when over 2GB data is written into `ChunkedByteBufferOutputStream`
Bug: Disable timeout for BroadcastQueryStageExec in AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/21 10:53
Affects Version/s: 3.0.3

This reverts SPARK-31475, as there are always more concurrent jobs running in AQE mode, especially when running multiple queries at the same time. Currently, the broadcast timeout does not record accurately for the BroadcastQueryStageExec only but also the time waiting for being scheduled. If all the resources are currently being occupied for materializing other stages, it timeouts without a chance to run actually.

 

!image-2021-08-04-18-53-44-879.png!

 

The default value is 300s, and it's hard to adjust the timeout for AQE mode. Usually, you need an extremely large number for real-world cases. As you can see the example, above, the timeout we used for it is 1800s, and obviously, it needs 3x more or something