av9ash/GitBugs huggingface dataset:
Bug: CostBasedJoinReorder can fail on self-join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/21 08:40
Affects Version/s: 3.2.0

 

For example:
{code:java}
test("join reorder with self-join") {
  val plan = t2.join(t1, Inner, Some(nameToAttr("t1.k-1-2") === nameToAttr("t2.k-1-5")))
    .select(nameToAttr("t1.v-1-10"))
    .join(t2, Inner, Some(nameToAttr("t1.v-1-10") === nameToAttr("t2.k-1-5")))

  // this can fail
  Optimize.execute(plan.analyze)
}
{code}
error:
{code:java}
[info]   java.lang.AssertionError: assertion failed
[info]   at scala.Predef$.assert(Predef.scala:208)
[info]   at org.apache.spark.sql.catalyst.optimizer.JoinReorderDP$.search(CostBasedJoinReorder.scala:178)
[info]   at org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder$.org$apache$spark$sql$catalyst$optimizer$CostBasedJoinReorder$$reorder(CostBasedJoinReorder.scala:64)
[info]   at org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder$$anonfun$1.applyOrElse(CostBasedJoinReorder.scala:45)
[info]   at org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder$$anonfun$1.applyOrElse(CostBasedJoinReorder.scala:41)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$1(TreeNode.scala:317)
[info]   at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:317)
[info]   at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDown(LogicalPlan.scala:29)
[info]   at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDown(AnalysisHelper.scala:171)
[info]   at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDown$(AnalysisHelper.scala:169)
[info]   at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDown(LogicalPlan.scala:29)
[info]   at org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder$.apply(CostBasedJoinReorder.scala:41)
[info]   at org.apache.spark.sql.catalyst.optimizer.CostBasedJoinReorder$.apply(CostBasedJoinReorder.scala:35)
{code}
Bug: Have functionality to trace back Spark SQL queries from the application ID that got submitted on YARN
Status: Open | Priority: Major | Resolution: 
Created: 03/Feb/21 11:46
Affects Version/s: 1.6.3

We need to have Application Id from resource manager mapped to the specific spark sql query that got executed with respect to that application Id so that back tracing is possible.

For example : if i run a query using spark shell : 

spark.sql("select dt.d_year,item.i_brand_id brand_id,item.i_brand brand,sum(ss_ext_sales_price) sum_agg from date_dim dt,store_sales,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manufact_id = 436 and dt.d_moy=12 group by dt.d_year,item.i_brand,item.i_brand_id order by dt.d_year,sum_agg desc,brand_id limit 100").show();

When  i see the event logs or the history server i don't see the query anywhere, but the query plan is there, so it becomes difficult to trace back what query actually got submitted. (if have to map it to the specific application Id on yarn)
Bug: Support regexp as a function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/21 09:00
Affects Version/s: 2.4.7

We have equality in SqlBase.g4 for RLIKE: 'RLIKE' | 'REGEXP';
We seemed to miss adding REGEXP as a SQL function just like RLIKE

This is also registered in Hive as a function, we can reduce the migration pain for those users
Bug: Expose the number of truncated paths in Utils.buildLocationMetadata()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/21 05:01
Affects Version/s: 3.2.0

SPARK-31793 introduces Utils.buildLocationMetadata() to reduce the length of location metadata. It effectively reduced the memory usage as only a few paths will be included (and it's controlled by threshold value), but there's no indication of the fact for truncation.

If the first 2 of 5 paths are only fit to the threshold, Utils.buildLocationMetadata() shows the first 2 paths, but there's no mention that 3 paths are truncated. Even no mention that some paths have been truncated; it shows the same with the output just first 2 paths were presented. This could bring confusion.

We should allow more space (like 10+ chars) to represent the fact N paths are truncated.
Bug: toPandas failed with error: buffer source array is read-only when Arrow with self-destruct is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/21 13:11
Affects Version/s: 3.2.0

Environment:

apache/spark master 
 pandas version > 1.0.5

Reproduce code:
{code:java}
spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
spark.conf.set('spark.sql.execution.arrow.pyspark.selfDestruct.enabled', True)
spark.createDataFrame(sc.parallelize([(i,) for i in range(13)], 1), 'id long').selectExpr('IF(id % 3==0, id+1, NULL) AS f1', '(id+1) % 2 AS label').toPandas()['label'].value_counts()
{code}
Get error like:
{quote}Traceback (most recent call last): 
 File "<stdin>", line 1, in <module>
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/core/base.py", line 1033, in value_counts
 dropna=dropna,
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/core/algorithms.py", line 820, in value_counts
 keys, counts = value_counts_arraylike(values, dropna)
 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pandas/core/algorithms.py", line 865, in value_counts_arraylike
 keys, counts = f(values, dropna)
 File "pandas/_libs/hashtable_func_helper.pxi", line 1098, in pandas._libs.hashtable.value_count_int64
 File "stringsource", line 658, in View.MemoryView.memoryview_cwrapper
 File "stringsource", line 349, in View.MemoryView.memoryview.__cinit__
 ValueError: buffer source array is read-only
{quote}
Bug: Checking duplicate static partition columns doesn't respect case sensitive conf
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/21 01:10
Affects Version/s: 2.4.7

When parsing the partition spec, Spark will call `org.apache.spark.sql.catalyst.parser.ParserUtils.checkDuplicateKeys` to check if there are duplicate partition column names in the list. But this method is always case sensitive and doesn't discover duplicate partition column names when using different cases.
Bug: Self-join after cogroup applyInPandas fails due to unresolved conflicting attributes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/21 02:53
Affects Version/s: 3.0.0

 
{code:java}
df = spark.createDataFrame([(1, 1)], ("column", "value"))row = df.groupby("ColUmn").cogroup(
    df.groupby("COLUMN")
).applyInPandas(lambda r, l: r + l, "column long, value long")
row.join(row).show()
{code}
{code:java}
Conflicting attributes: column#163321L,value#163322L
;;
’Join Inner
:- FlatMapCoGroupsInPandas [ColUmn#163312L], [COLUMN#163312L], <lambda>(column#163312L, value#163313L, column#163312L, value#163313L), [column#163321L, value#163322L]
:  :- Project [ColUmn#163312L, column#163312L, value#163313L]
:  :  +- LogicalRDD [column#163312L, value#163313L], false
:  +- Project [COLUMN#163312L, column#163312L, value#163313L]
:     +- LogicalRDD [column#163312L, value#163313L], false
+- FlatMapCoGroupsInPandas [ColUmn#163312L], [COLUMN#163312L], <lambda>(column#163312L, value#163313L, column#163312L, value#163313L), [column#163321L, value#163322L]
   :- Project [ColUmn#163312L, column#163312L, value#163313L]
   :  +- LogicalRDD [column#163312L, value#163313L], false
   +- Project [COLUMN#163312L, column#163312L, value#163313L]
      +- LogicalRDD [column#163312L, value#163313L], false
{code}
Bug: Upgrade aws kinesis to 1.14.0 and java sdk 1.11.844
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Feb/21 05:54
Affects Version/s: 3.2.0

Upgrade aws kinesis and java sdk to support to catch up minimum requirement for IAM role for service accounts: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
Bug: generate-contributors.py, releaseutils.py and translate-contributors.py are broken
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/21 08:36
Affects Version/s: 3.2.0

This is broken after SPARK-32719. {{raw_input}} does not exist in {{releaseutils}} but in Python 2:

{code}
Traceback (most recent call last):
  File "dev/create-release/generate-contributors.py", line 25, in <module>
    from releaseutils import tag_exists, raw_input, get_commits, yesOrNoPrompt, get_date, \
ImportError: cannot import name 'raw_input' from 'releaseutils' (/.../spark/dev/create-release/releaseutils.py)
{code}

In addition, these scripts look only working with Python 2. We should drop Python 2.
Bug: JDBC connection provider is not removing kerberos credentials from JVM security context
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/21 13:43
Affects Version/s: 3.0.2

Some of the built-in JDBC connection providers are changing the JVM security context to do the authentication which is fine. The problematic part is that executors can be reused by another query. The following situation leads to incorrect behaviour:
 * Query1 opens JDBC connection and changes JVM security context in Executor1
 * Query2 tries to open JDBC connection but it realizes there is already an entry for that DB type in Executor1
 * Query2 is not changing JVM security context and uses Query1 keytab and principal
 * Query2 fails with authentication error
Bug: HiveThriftServer2 startWithContext may hang with a race issue 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/21 02:19
Affects Version/s: 3.0.1

```
21:43:26.809 WARN org.apache.thrift.server.TThreadPoolServer: Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: No underlying server socket.
	at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:126)
	at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
	at org.apache.thrift.transport.TServerTransport.acceException in thread "Thread-15" java.io.IOException: Stream closed
	at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at scala.sys.process.BasicIO$.loop$1(BasicIO.scala:238)
	at scala.sys.process.BasicIO$.transferFullyImpl(BasicIO.scala:246)
	at scala.sys.process.BasicIO$.transferFully(BasicIO.scala:227)
	at scala.sys.process.BasicIO$.$anonfun$toStdOut$1(BasicIO.scala:221)
```
the TServer might try to serve even the stop is called
Bug: Upgrade Jetty to fix CVE-2020-27218
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/21 03:22
Affects Version/s: 2.4.7

CVE-2020-27218 affects the currently used Jetty 9.4.34 so let's upgrade it.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-27218.
Bug: Update Spark SQL guide about rebase DS options and SQL configs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/21 15:16
Affects Version/s: 3.2.0

Describe the following SQL configs:
* spark.sql.legacy.parquet.int96RebaseModeInWrite
* spark.sql.legacy.parquet.datetimeRebaseModeInWrite
* spark.sql.legacy.parquet.int96RebaseModeInRead
* spark.sql.legacy.parquet.datetimeRebaseModeInRead
* spark.sql.legacy.avro.datetimeRebaseModeInWrite
* spark.sql.legacy.avro.datetimeRebaseModeInRead

And Avro/Parquet options datetimeRebaseMode and int96RebaseMode.
Bug: Support spark.kubernetes.executor.disableConfigMap
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/21 20:51
Affects Version/s: 3.2.0

In shared Kubernetes clusters, Spark could be restricted from creating and deleting config maps in job namespaces.

It would be helpful if the current mandatory config map creation could be optional. User may still take responsibility of handing Spark conf files separately.
Bug: Loosen AvroSerializer validation to allow extra nullable user-provided fields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Feb/21 17:22
Affects Version/s: 3.0.1

Currently, when writing out Avro data using a custom schema ({{avroSchema}}), if there are any extra Avro fields which do not have a matching field in the Catalyst schema, the serialization will fail. This is much more strict than on the deserialization path, where Avro fields not present in the Catalyst schema are ignored, and Catalyst fields not present in the Avro schema are allowed as long as they are nullable. I believe it will be more user-friendly if extra Avro fields are allowed, as long as they are optional. This makes it easier for users to write out data with Avro schemas which may be outside of their control.

If there is concern about the safety of this approach (i.e. there are use cases where users want strict matching), we can make it configurable.
Bug: ANSI SQL: LATERAL derived table(T491)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/21 01:09
Affects Version/s: 3.0.0

Subqueries appearing in {{FROM}} can be preceded by the key word {{LATERAL}}. This allows them to reference columns provided by preceding {{FROM}} items. (Without {{LATERAL}}, each subquery is evaluated independently and so cannot cross-reference any other {{FROM}} item.)

Table functions appearing in {{FROM}} can also be preceded by the key word {{LATERAL}}, but for functions the key word is optional; the function's arguments can contain references to columns provided by preceding {{FROM}} items in any case.

A {{LATERAL}} item can appear at top level in the {{FROM}} list, or within a {{JOIN}} tree. In the latter case it can also refer to any items that are on the left-hand side of a {{JOIN}} that it is on the right-hand side of.

When a {{FROM}} item contains {{LATERAL}} cross-references, evaluation proceeds as follows: for each row of the {{FROM}} item providing the cross-referenced column(s), or set of rows of multiple {{FROM}} items providing the columns, the {{LATERAL}} item is evaluated using that row or row set's values of the columns. The resulting row(s) are joined as usual with the rows they were computed from. This is repeated for each row or set of rows from the column source table(s).

A trivial example of {{LATERAL}} is
{code:sql}
SELECT * FROM foo, LATERAL (SELECT * FROM bar WHERE bar.id = foo.bar_id) ss;
{code}

*Feature ID*: T491

[https://www.postgresql.org/docs/11/queries-table-expressions.html#QUERIES-FROM]
[https://github.com/postgres/postgres/commit/5ebaaa49445eb1ba7b299bbea3a477d4e4c0430]
Bug: spark.createDataFrame does not support Pandas StringDtype extension type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/21 15:02
Affects Version/s: 3.0.1

The following test case demonstrates the problem:
{code:java}
import pandas as pd
from pyspark.sql import SparkSession, types

spark = SparkSession.builder.appName(__file__)\
    .config("spark.sql.execution.arrow.pyspark.enabled","true") \
    .getOrCreate()

good = pd.DataFrame([["abc"]], columns=["col"])

schema = types.StructType([types.StructField("col", types.StringType(), True)])
df = spark.createDataFrame(good, schema=schema)

df.show()

bad = good.copy()
bad["col"]=bad["col"].astype("string")

schema = types.StructType([types.StructField("col", types.StringType(), True)])
df = spark.createDataFrame(bad, schema=schema)

df.show(){code}
The error:
{code:java}
C:\Python\3.8.3\lib\site-packages\pyspark\sql\pandas\conversion.py:289: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below:
  Cannot specify a mask or a size when passing an object that is converted with the __arrow_array__ protocol.
Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true.
  warnings.warn(msg)
{code}
Bug: Unify v1 and v2 ALTER NAMESPACE .. SET LOCATION tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Feb/21 15:10
Affects Version/s: 3.2.0

Extract ALTER NAMESPACE .. SET LOCATION tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Update Spark Create Table DDL Docs
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Feb/21 17:18
Affects Version/s: 3.0.3

Within the `CREATE TABLE` docs, the `OPTIONS` and `TBLPROPERTIES`specify `key=value` parameters with a `=` as the delimiter between the key value pairs. 
The `=` is optional and can be space delimited. We should document that both methods are supported when defining these parameters.

 

One location within the current docs page that should be updated: 

[https://spark.apache.org/docs/latest/sql-ref-syntax-ddl-create-table-datasource.html]

 

Code reference showing equal as an optional parameter:

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L401
Bug: Move data source options docs for Python and Scala into a single page
Status: Resolved | Priority: Major | Resolution: Done
Created: 22/Feb/21 02:43
Affects Version/s: 3.2.0

Currently we documents options of various data sources for Python and Scala in a separated page.
 * Python [here|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=registerjava#pyspark.sql.DataFrameReader]
 * Scala [here|https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameReader.html]

However, we'd better to move this into a single page so that we can fix only one place when we need to manage the data source options. See [Avro|https://spark.apache.org/docs/latest/sql-data-sources-avro.html#apache-avro-data-source-guide] for example.

Also, we need to add missing "CSV Files" and "TEXT Files" page for [Data Sources documents|https://spark.apache.org/docs/latest/sql-data-sources.html#data-sources].

To clarify the task,
 # create "CSV Files" page for Data Source documents.
 # create "TEXT files" page for Data Source documents.
 # move Parquet data source options from Python and Scala into a single page.
 # move other data source options from Python and Scala into a single page.
Bug: my personal blog 
Status: Closed | Priority: Major | Resolution: Abandoned
Created: 21/Feb/21 05:18
Affects Version/s: 3.0.2

Hello dear!

Hello dear!

i am prabhu From Barmer rajasthan 

i love share new information with people 

 

 

Please read My Article
 # [What is operating system in hindi full information|https://www.prabhubarmer.in/operating-system-kya-hai/]
 # [What is blogging in hindi . how to earn money form blogging|https://www.prabhubarmer.in/what-is-blogging-in-hindi/]
 # [Web server kya hota hai ? what is web server in hindi|https://www.prabhubarmer.in/what-is-web-server-in-hindi/]
 # [Computer kya hai ? kitane prakar ka hota hai ?|https://www.prabhubarmer.in/types-of-computer/]
 # [Domain name kya hai ? domain kitne prkar ke hote hai ?|https://www.prabhubarmer.in/domain-name-kya-hai/]
 # [How to grow hair using home tips . grow hair fast in hindi|https://www.prabhubarmer.in/hair-growth-tips-in-hindi-at-home/]
 # [Mutual fund kya hai ? full inforamtion about mutual fund in hindi|https://www.prabhubarmer.in/mutual-fund-kya-hai/]
 # [UPSC kya hai ? UPSC Full Form in hindi|https://www.prabhubarmer.in/upsc-full-form-in-hindi-full-information/]
 # [SSC kya hai ? Full Information Of SSC ? SSC Full Form|https://www.prabhubarmer.in/ssc-full-form/]
 # [how to update adhar card online ? 3 way to update aadhar card|https://www.prabhubarmer.in/how-to-update-]
 # [Haldi ke fayade in hindi ? termaric benifits home remedits|https://www.prabhubarmer.in/haldi-ke-fayde-in-hindi/]
 # [Seo kya hai ? seo kaise kare ? type of seo in hindi|https://www.prabhubarmer.in/seo-in-hindi-full-information/]
 # [On Page Seo In Hindi ? On Page Seo Kya hai ?|https://www.prabhubarmer.in/on-page-seo-in-hindi/]
 # [Signal app kya hai full informtion in hindi|https://www.prabhubarmer.in/signal-app-kya-hai]

 

 

 

Profile Links

[https://in.pinterest.com/prabhuBarmer/]

[https://www.facebook.com/DesignerPrabhu/]

[https://www.instagram.com/prabhu_barmer/]

[https://www.blogger.com/profile/05357736552959285032]

[https://profiles.wordpress.org/prabhubarmer/]

[https://github.com/PrabhuBarmer]

[https://twitter.com/Prabhu_Barmer]

[http://www.icoph.org/connections/user_profile/0033o00003vcVGgAAM/Prabhu-Prabhu.html]

[https://www.misterpoll.com/users/756839]

[https://try.gitea.io/123prabhu_bmr]

[https://hub.docker.com/u/prabhubarmer]

[https://pastebin.com/u/Prabhubarmer]
Bug: Support configurable Avro schema field matching for positional or by-name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/21 18:58
Affects Version/s: 3.0.1

When reading an Avro dataset (using the dataset's schema or by overriding it with 'avroSchema') or writing an Avro dataset with a provided schema by 'avroSchema', currently the matching of Catalyst-to-Avro fields is done by field name.

This behavior is somewhat recent; prior to SPARK-27762 (fixed in 3.0.0), at least on the write path, we would match the schemas by positionally ("structural" comparison). While I agree that this is much more sensible for default behavior, I propose that we make this behavior configurable using an {{option}} for the Avro datasource. Even at the time that SPARK-27762 was handled, there was [interest in making this behavior configurable|https://github.com/apache/spark/pull/24635#issuecomment-494205251], but it appears it went unaddressed.

There is precedence for configurability of this behavior as seen in SPARK-32864, which added this support for ORC. Besides this precedence, the behavior of Hive is to perform matching positionally ([ref|https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-WritingtablestoAvrofiles]), so this is behavior that Hadoop/Hive ecosystem users are familiar with:
{quote}
Hive is very forgiving about types: it will attempt to store whatever value matches the provided column in the equivalent column position in the new table. No matching is done on column names, for instance.
{quote}
Bug: When hit  ApplicationAttemptNotFoundException, we can't just stop app for all case
Status: In Progress | Priority: Major | Resolution: 
Created: 02/Feb/21 12:13
Affects Version/s: 3.2.0

We always meet case that because yarn queue's setting, some app's container is preempted by a higher level request due to the scheduling framework.

 

In this case Spark just stoped immediately. This case we can have a retry.