av9ash/GitBugs huggingface dataset:
Bug: Wrong semantics for null IN (empty list) - IN expression execution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/23 03:28
Affects Version/s: 3.4.0

{{null IN (empty list)}} incorrectly evaluates to null, when it should evaluate to false. (The reason it should be false is because a IN (b1, b2) is defined as a = b1 OR a = b2, and an empty IN list is treated as an empty OR which is false. This is specified by ANSI SQL.)

Many places in Spark execution (In, InSet, InSubquery) and optimization (OptimizeIn, NullPropagation) implemented this wrong behavior. Also note that the Spark behavior for the null IN (empty list) is inconsistent in some places - literal IN lists generally return null (incorrect), while IN/NOT IN subqueries mostly return false/true, respectively (correct) in this case.

This is a longstanding correctness issue which has existed since null support for IN expressions was first added to Spark.

Doc with more details: [https://docs.google.com/document/d/1k8AY8oyT-GI04SnP7eXttPDnDj-Ek-c3luF2zL6DPNU/edit]
Bug: Install different Python linter dependencies for daily testing of different Spark versions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/23 06:10
Affects Version/s: 4.0.0

Fix daily test python lint check failure for branches 3.3 and 3.4

 

3.4 : [https://github.com/apache/spark/actions/runs/5654787844/job/15318633266]

3.3 : https://github.com/apache/spark/actions/runs/5653655970/job/15315236052
Bug: Fix AssertionError when converting UDTF output to a complex type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/23 23:41
Affects Version/s: 3.5.0

{code:java}
class TestUDTF:
    def eval(self):
        yield {'a': 1, 'b': 2},

udtf(TestUDTF, returnType="x: map<string,int>")().show() {code}
This will fail with:

  File "pandas/_libs/lib.pyx", line 2834, in pandas._libs.lib.map_infer
  File "python/pyspark/sql/pandas/types.py", line 804, in convert_map
    assert isinstance(value, dict)
AssertionError

Same for `convert_struct`
Bug: Migrate PySpark tests to use PySpark built-in test utils
Status: Open | Priority: Major | Resolution: 
Created: 28/Jul/23 16:39
Affects Version/s: 3.5.0

The Jira ticket SPARK-44042 SPIP: PySpark Test Framework introduces a new PySpark test framework. Some of the user-facing testing util APIs include assertDataFrameEqual, assertSchemaEqual, and assertPandasOnSparkEqual.

With the new testing framework, we should migrate old tests in the Spark codebase to use the new testing utils.
Bug: Add support for pandas-on-Spark DataFrame assertDataFrameEqual
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/23 21:18
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Improve error messages for Python UDTF arrow type casts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/23 21:35
Affects Version/s: 3.5.0

Currently, if a Python UDTF outputs a type that is incompatible with the specified output schema, Spark will throw the following confusing error message:
{code:java}
  File "pyarrow/array.pxi", line 1044, in pyarrow.lib.Array.from_pandas
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert [1, 2] with type list: tried to convert to int32{code}
We should improve this.
Bug: Make `hive-thriftserver` module daily test pass
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/23 02:22
Affects Version/s: 4.0.0

[https://github.com/LuciferYang/spark/actions/runs/5694334367/job/15435297305]

 
{code:java}
*** RUN ABORTED ***
20159  java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
20160  at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
20161  at java.lang.Class.forName0(Native Method)
20162  at java.lang.Class.forName(Class.java:348)
20163  at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClassInternal(GenericUDFBridge.java:142)
20164  at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
20165  at org.apache.hadoop.hive.ql.exec.FunctionInfo.getFunctionClass(FunctionInfo.java:151)
20166  at org.apache.hadoop.hive.ql.exec.Registry.addFunction(Registry.java:519)
20167  at org.apache.hadoop.hive.ql.exec.Registry.registerUDF(Registry.java:163)
20168  at org.apache.hadoop.hive.ql.exec.Registry.registerUDF(Registry.java:154)
20169  at org.apache.hadoop.hive.ql.exec.Registry.registerUDF(Registry.java:147)
20170  ...
20171  Cause: java.lang.ClassNotFoundException: org.codehaus.jackson.map.type.TypeFactory
20172  at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
20173  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
20174  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
20175  at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
20176  at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
20177  at java.lang.Class.forName0(Native Method)
20178  at java.lang.Class.forName(Class.java:348)
20179  at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClassInternal(GenericUDFBridge.java:142)
20180  at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:132)
20181  at org.apache.hadoop.hive.ql.exec.FunctionInfo.getFunctionClass(FunctionInfo.java:151)
20182  ... {code}
Bug: Add option for thread pool to perform maintenance for RocksDB/HDFS State Store Providers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/23 00:57
Affects Version/s: 3.5.0

Maintenance tasks on StateStore was being done by a single background thread, which is prone to straggling. In this change, the single background thread would instead schedule maintenance tasks to a thread pool.
Introduce {{spark.sql.streaming.stateStore.enableStateStoreMaintenanceThreadPool}} config so that the user can enable a thread pool for maintenance manually.
Introduce {{spark.sql.streaming.stateStore.numStateStoreMaintenanceThreads}} config so the thread pool size is configurable.
Bug: Hive Generic UDF support no longer supports short-circuiting of argument evaluation
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/23 22:06
Affects Version/s: 3.5.0

PR [https://github.com/apache/spark/pull/39555] changed DeferredObject to no longer contain a function, and instead contains a value. This removes the deferred evaluation capability and means that HiveGenericUDF implementations can no longer short-circuit the evaluation of their arguments, which could be a performance issue for some users.

Here is a relevant javadoc comment from the Hive source for DeferredObject:

{code:java}
  /**
   * A Defered Object allows us to do lazy-evaluation and short-circuiting.
   * GenericUDF use DeferedObject to pass arguments.
   */
  public static interface DeferredObject {
{code}
Bug: UDF should support function taking value classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/23 12:46
Affects Version/s: 3.4.1

Running the following code in a spark 
```
final case class ValueClass(a: Int) extends AnyVal
final case class Wrapper(v: ValueClass)

val f = udf((a: ValueClass) => a.a > 0)

spark.createDataset(Seq(Wrapper(ValueClass(1)))).filter(f(col("v"))).show()
```

fails with
```
java.lang.ClassCastException: class org.apache.spark.sql.types.IntegerType$ cannot be cast to class org.apache.spark.sql.types.StructType (org.apache.spark.sql.types.IntegerType$ and org.apache.spark.sql.types.StructType are in unnamed module of loader 'app')
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveEncodersInUDF$$anonfun$apply$42$$anonfun$applyOrElse$218.$anonfun$applyOrElse$220(Analyzer.scala:3241)
  at scala.Option.map(Option.scala:242)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveEncodersInUDF$$anonfun$apply$42$$anonfun$applyOrElse$218.$anonfun$applyOrElse$219(Analyzer.scala:3239)
  at scala.collection.immutable.List.map(List.scala:246)
  at scala.collection.immutable.List.map(List.scala:79)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveEncodersInUDF$$anonfun$apply$42$$anonfun$applyOrElse$218.applyOrElse(Analyzer.scala:3237)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveEncodersInUDF$$anonfun$apply$42$$anonfun$applyOrElse$218.applyOrElse(Analyzer.scala:3234)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUpWithPruning$2(TreeNode.scala:566)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformUpWithPruning(TreeNode.scala:566)
```
Bug: spark 3.2+ can not read hive table with hbase serde when hbase StorefileSize  is 0
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 30/Jul/23 03:01
Affects Version/s: 3.2.3

We using spark to read a hive table with hbase serde . We found that when the hbase table data is relatively small (hbase StorefileSize is 0), the data read by spark 3.2 or 3.5 is empty, and there is no error message.

But when using spark2.4 or hive to read, the data can be read normally. Other information shows that spark3.1 can also read data normally, can anyone provide some ideas?
Bug: Add jobTags to SparkListenerSQLExecutionStart
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/23 22:30
Affects Version/s: 3.5.0

As part SPARK-43952, users can define job tags via SparkContext.addJobTag. These tags are then used to trigger cancelation via SparkContext.cancelJobsByTag. Furthermore, these tags can be used to logically group multiple jobs together.

Listener of job events can retrieve job tags via SparkListenerJobStart.props.getProperty(SparkContext.SPARK_JOB_TAGS)

Listener of SQL events can link SparkListenerJobStart & SparkListenerSQLExecutionStart via SparkListenerJobStart.props.getProperty(SQLExecution.EXECUTION_ID_KEY).

However, some SQL executions do not trigger jobs (i.e. commands). As such listeners of SQL executions cannot resolve job tags of all executions.
Bug: Use jobTags in SparkListenerSQLExecutionStart to get SQL Execution ID for Spark UI Connect page
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/23 16:50
Affects Version/s: 3.5.0

Follow up to https://issues.apache.org/jira/browse/SPARK-44394 and https://issues.apache.org/jira/browse/SPARK-44591. Use jobTags in SparkListenerSQLExecutionStart to get the corresponding SQL Execution for Spark Connect request rather than SparkListenerJobStart.props.getProperty(SQLExecution.EXECUTION_ID_KEY), which won't work when the SQL Execution does not trigger a job.
Bug:     Define the computing logic through  PartitionEvaluator API and use it in RowToColumnarExec & ColumnarToRowExec SQL operators.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/23 00:40
Affects Version/s: 3.5.0

   
Define the computing logic through PartitionEvaluator API and use it in RowToColumnarExec & ColumnarToRowExec SQL operators.
Bug: Fix pandas-on-Spark type checks for assertDataFrameEqual
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/23 01:44
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: Terminate streaming queries when a session times out in Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/23 17:51
Affects Version/s: 3.4.1

Spark Connect server keeps a session id at connect server alive as long as a streaming query is alive. This may not be desirable. If the client disconnects long enough and times out in server, we should let the session mapping to be removed and terminate the streaming queries.
Bug: Add a new Spark UI page for Spark Connect 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/23 17:18
Affects Version/s: 3.5.0

Add a new Spark UI page to display session and execution information for Spark Connect. This builds of the work in SPARK-43923 (https://github.com/apache/spark/pull/41443) that adds the relevant SparkListenerEvents.
Bug: Remove useless private object `ParseState` from `IntervalUtils`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/23 05:03
Affects Version/s: 4.0.0

SPARK-44326（https://github.com/apache/spark/pull/41885/files#diff-e695bcf363031df04cc16c96f1b62fb35093807ee1157c3cbecae3cccb9a3f4e） moved the relevant code from `IntervalUtils` to `SparkIntervalUtils` (including the definition of another `private object ParseState`), but did not delete the definition of `private object ParseState` in `IntervalUtils`.
Bug: Support Python UDTFs with empty schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/23 21:21
Affects Version/s: 3.5.0

Support UDTFs with empty schema, for example:

{code:python}
>>> class TestUDTF:
...   def eval(self):
...     yield tuple()
{code}

Currently it fails with `useArrow=True`:

{code:python}
>>> udtf(TestUDTF, returnType=StructType())().collect()
Traceback (most recent call last):
...
ValueError: not enough values to unpack (expected 2, got 0)
{code}

whereas without Arrow:

{code:python}
>>> udtf(TestUDTF, returnType=StructType(), useArrow=False)().collect()
[Row()]
{code}

Otherwise, we should raise an error without Arrow, too, to be consistent.
Bug: DataSource v2 Scans should not require planning the input partitions on explain
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/23 21:55
Affects Version/s: 3.4.0

Right now, we will always call `planInputPartitions()` for a DSv2 implementation even if there is no spark job run but only explain.

We should provide a way to avoid scanning all input partitions just to determine if the input is columnar or not. The scan should provide an override.
Bug: Spark CI Improvement
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/23 11:00
Affects Version/s: 4.0.0

I have an offline discussion with [~gurwls223] and [~LuciferYang], and we think that several points should be improved:

# it should be tested with Maven
# all supported Python Versions should be tested
# clean up unused files ASAP, since the testing resource is quite limited


To avoid increase the workload too much, we can add daily GA first.
Bug: Spark Connect DataFrame does not allow to add custom instance attributes and check for it
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/23 20:32
Affects Version/s: 3.4.1

```
df = spark.range(10)
df._test = 10

assert(hasattr(df, "_test"))
assert(!hasattr(df, "_test_no"))
```

Treats `df._test` like a column
Bug: Wrong semantics for null IN (empty list) - optimization rules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/23 16:27
Affects Version/s: 3.4.0

{{null IN (empty list)}} incorrectly evaluates to null, when it should evaluate to false. (The reason it should be false is because a IN (b1, b2) is defined as a = b1 OR a = b2, and an empty IN list is treated as an empty OR which is false. This is specified by ANSI SQL.)

Many places in Spark execution (In, InSet, InSubquery) and optimization (OptimizeIn, NullPropagation) implemented this wrong behavior. Also note that the Spark behavior for the null IN (empty list) is inconsistent in some places - literal IN lists generally return null (incorrect), while IN/NOT IN subqueries mostly return false/true, respectively (correct) in this case.

This is a longstanding correctness issue which has existed since null support for IN expressions was first added to Spark.

Doc with more details: [https://docs.google.com/document/d/1k8AY8oyT-GI04SnP7eXttPDnDj-Ek-c3luF2zL6DPNU/edit]
Bug: Add a dev utility to generate PySpark tests with LLM
Status: Open | Priority: Major | Resolution: 
Created: 25/Jul/23 14:59
Affects Version/s: 3.5.0

h2. Summary

This ticket adds a dev utility script to help generate PySpark tests using LLM response. The purpose of this experimental script is to encourage PySpark developers to test their code thoroughly, to avoid introducing regressions in the codebase. Historically, PySpark has had code regressions due to insufficient testing of public APIs.

Below, we outline some common edge case scenarios for PySpark DataFrame APIs, from the perspective of arguments. Many of these edge cases are passed into the LLM through the script's base prompt.

Please note that this list is not exhaustive, but rather a starting point. Some of these cases may not apply, depending on the situation. We encourage all PySpark developers to carefully consider edge case scenarios when writing tests.
h2. Table of Contents
 # None
 # Ints
 # Floats
 # Strings
 # Single column / column name
 # Multi column / column names
 # DataFrame argument

h3. 1. None
 * Empty input
 * None type

h3. 2. Ints
 * Negatives
 * 0
 * value > Int.MaxValue
 * value < Int.MinValue

h3. 3. Floats
 * Negatives
 * 0.0
 * Float(“nan”)
 * Float("inf")
 * Float("-inf")
 * decimal.Decimal
 * numpy.float16

h3. 4. Strings
 * Special characters
 * Spaces
 * Empty strings

h3. 5. Single column / column name
 * Non-existent column
 * Empty column name
 * Column name with special characters, e.g. dots
 * Multi columns with the same name
 * Nested column vs. quoted column name, e.g. ‘a.b.c’ vs ‘`a.b.c`’
 * Column of special types, e.g. nested type;
 * Column containing special values, e.g. Null;

h3. 6. Multi column / column names
 * Empty input; e.g DataFrame.drop()
 * Special cases for each single column
 * Mix column with column names; e.g. DataFrame.drop(“col1”, df.col2, “col3”)
 * Duplicated columns; e.g. DataFrame.drop(“col1”, col(“col1”))

h3. 7. DataFrame argument
 * DataFrame argument
 * Empty dataframe; e.g. spark.range(5).limit(0)
 * Dataframe with 0 columns, e.g. spark.range(5).drop('id')
 * Dataset with repeated arguments
 * Local dataset (pd.DataFrame) containing unsupported datatype
Bug: Clarify error for unsupported arg data type in assertDataFrameEqual
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/23 01:52
Affects Version/s: 3.5.0

SPIP: https://docs.google.com/document/d/1OkyBn3JbEHkkQgSQ45Lq82esXjr9rm2Vj7Ih_4zycRc/edit#heading=h.f5f0u2riv07v
Bug: It's impossible to get column by index if names are not unique
Status: Open | Priority: Major | Resolution: 
Created: 25/Jul/23 14:18
Affects Version/s: 3.4.0

So, I have a dataframe with non-unique columns names:

 
{code:java}
df = spark_session.createDataFrame([[1,2,3], [4,5,6]], ['a', 'a', 'c']) {code}
 

It works fine.

 

Now I try to get a column with non-unique name by index
{code:java}
df[0]
{code}
Expectation: It returns first of the columns

Note, the [doc|[https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.__getitem__.html]] doesn't mention non-unique name as a precondition.

 

Actual result:

It throws exception:

 
{noformat}
AnalysisException                         Traceback (most recent call last)
Cell In[71], line 1
----> 1 df[0]

File /usr/local/spark/python/pyspark/sql/dataframe.py:2935, in DataFrame.__getitem__(self, item)
   2933     return self.select(*item)
   2934 elif isinstance(item, int):
-> 2935     jc = self._jdf.apply(self.columns[item])
   2936     return Column(jc)
   2937 else:

File /usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\
   1317     self.command_header +\
   1318     args_command +\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File /usr/local/spark/python/pyspark/errors/exceptions/captured.py:175, in capture_sql_exception.<locals>.deco(*a, **kw)
    171 converted = convert_exception(e.java_exception)
    172 if not isinstance(converted, UnknownException):
    173     # Hide where the exception came from that shows a non-Pythonic
    174     # JVM exception message.
--> 175     raise converted from None
    176 else:
    177     raise

AnalysisException: [AMBIGUOUS_REFERENCE] Reference `a` is ambiguous, could be: [`a`, `a`].{noformat}
Bug: Move commands to CTEDef code path and deprecate CTE inline path
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/23 17:11
Affects Version/s: 3.5.0

Right now our CTE resolution code path is diverged: query with commands go into CTE inline code path where non-command queries go into CTEDef code path (see https://github.com/apache/spark/blob/42719d9425b9a24ef016b5c2874e522b960cf114/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala#L50 ).

For longer term we should migrate command queries go through CTEDef as well and deprecate the CTE inline path.
Bug: Cleanup .spark-staging directories when yarn application fails
Status: Open | Priority: Major | Resolution: 
Created: 25/Jul/23 12:21
Affects Version/s: 3.4.1

Spark creates the staging directories like .hive-staging, .spark-staging etc which get created when you run an dynamic insert overwrite to a partitioned table. Spark spends maximum time in renaming the partitioned files, and because GCS renaming are too slow, there are frequent scenarios where YARN fails due to network error etc.

Such directories will remain forever in Google Cloud Storage, in case the yarn application manager gets killed.
 
Over time this pileup and incurs a lot of cloud storage cost.
 
Can we update our File committer to clean up the temporary directories in case the job commit fails.

PS : This request is specifically for GCS.
Image for reference 
 

!image-2023-07-25-17-52-55-006.png|width=458,height=177!
Bug: Implement PyArrow `self_destruct` feature for `toPandas`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/23 00:09
Affects Version/s: 4.0.0

Implement PyArrow `self_destruct` feature for `toPandas`

To make the Spark configuration `spark.sql.execution.arrow.pyspark.selfDestruct.enabled` be used to enable PyArrow’s `self_destruct` feature in Spark Connect, which can save memory when creating a Pandas DataFrame via `toPandas` by freeing Arrow-allocated memory while building the Pandas DataFrame.
Bug: parse_url treats key as regular expression
Status: Open | Priority: Major | Resolution: 
Created: 20/Jul/23 14:36
Affects Version/s: 3.2.0

To be clear I am not 100% sure that this is a bug. It might be a feature, but I don't see anywhere that it is used as a feature. If it is a feature it really should be documented, because there are pitfalls. If it is a bug it should be fixed because it is really confusing and it is simple to shoot yourself in the foot.

```scala
> val urls = Seq("http://foo/bar?abc=BAD&a.c=GOOD", "http://foo/bar?a.c=GOOD&abc=BAD").toDF
> urls.selectExpr("parse_url(value, 'QUERY', 'a.c')").show(false)

+----------------------------+
|parse_url(value, QUERY, a.c)|
+----------------------------+
|BAD                         |
|GOOD                        |
+----------------------------+

> urls.selectExpr("parse_url(value, 'QUERY', 'a[c')").show(false)
java.util.regex.PatternSyntaxException: Unclosed character class near index 15
(&|^)a[c=([^&]*)
               ^
  at java.util.regex.Pattern.error(Pattern.java:1969)
  at java.util.regex.Pattern.clazz(Pattern.java:2562)
  at java.util.regex.Pattern.sequence(Pattern.java:2077)
  at java.util.regex.Pattern.expr(Pattern.java:2010)
  at java.util.regex.Pattern.compile(Pattern.java:1702)
  at java.util.regex.Pattern.<init>(Pattern.java:1352)
  at java.util.regex.Pattern.compile(Pattern.java:1028)

```

The simple fix is to quote the key when making the pattern.

```scala
  private def getPattern(key: UTF8String): Pattern = {
    Pattern.compile(REGEXPREFIX + Pattern.quote(key.toString) + REGEXSUBFIX)
  }
```