av9ash/GitBugs huggingface dataset:
Bug: Add support for reading ORC/Parquet files with SymlinkTextInputFormat
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Jul/20 16:43
Affects Version/s: 3.1.0

Hive style symlink (SymlinkTextInputFormat) is commonly used in different analytic engines including prestodb and prestosql.

Currently SymlinkTextInputFormat works with JSON/CSV files but does not work with ORC/Parquet files in Apache Spark (and Apache Hive).

On the other hand, prestodb and prestosql support SymlinkTextInputFormat with ORC/Parquet files.

This issue is to add support for reading ORC/Parquet files with SymlinkTextInputFormat in Apache Spark.

 

Related links
 * Hive's SymlinkTextInputFormat: [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/SymlinkTextInputFormat.java]
 * prestosql's implementation to add support for reading avro files with SymlinkTextInputFormat: [https://github.com/vincentpoon/prestosql/blob/master/presto-hive/src/main/java/io/prestosql/plugin/hive/BackgroundHiveSplitLoader.java]
Bug: SessionState leaks SparkListener with multiple SparkSession
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 03/Jul/20 03:31
Affects Version/s: 3.0.0

Copied from [https://github.com/apache/spark/pull/28128#issuecomment-653102770]

I'd like to point out that this pr (https://github.com/apache/spark/pull/28128) doesn't fix the memory leaky completely. Once {{SessionState}} is touched, it will add two more listeners into the SparkContext, namely {{SQLAppStatusListener}} and {{ExecutionListenerBus}}

It can be reproduced easily as
{code:java}
  test("SPARK-31354: SparkContext only register one SparkSession ApplicationEnd listener") {
    val conf = new SparkConf()
      .setMaster("local")
      .setAppName("test-app-SPARK-31354-1")
    val context = new SparkContext(conf)
    SparkSession
      .builder()
      .sparkContext(context)
      .master("local")
      .getOrCreate()
      .sessionState // this touches the sessionState
    val postFirstCreation = context.listenerBus.listeners.size()
    SparkSession.clearActiveSession()
    SparkSession.clearDefaultSession()

    SparkSession
      .builder()
      .sparkContext(context)
      .master("local")
      .getOrCreate()
      .sessionState // this touches the sessionState
    val postSecondCreation = context.listenerBus.listeners.size()
    SparkSession.clearActiveSession()
    SparkSession.clearDefaultSession()
    assert(postFirstCreation == postSecondCreation)
  }
{code}
The problem can be reproduced by the above code.
Bug: UDF parse_url with a URL that contains pipe(|) will give incorrect result
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Jul/20 08:50
Affects Version/s: 2.4.4

I try to get the host from the URL with the code below.
{code:java}
Seq(
  "https://a.b.c/index.php?params1=a|b&params2=x",
  "https://a.b.c/index.php?params1=a"
)
  .toDF("url")
  .withColumn("host", callUDF("parse_url", $"url", lit("HOST")))
  .show(false){code}
The result of the code is as follows.
{code:java}
+---------------------------------------------+-----+
|url                                          |host |
+---------------------------------------------+-----+
|https://a.b.c/index.php?params1=a|b&params2=x|null |
|https://a.b.c/index.php?params1=a            |a.b.c|
+---------------------------------------------+-----+
{code}
It seems like the host becomes null when the URL contains any pipe(|) character.
Bug: Drop references to Master
Status: Open | Priority: Major | Resolution: 
Created: 16/Jul/20 14:55
Affects Version/s: 3.0.0

We have a lot of references to "master" in the code base. It will be beneficial to remove references to problematic language that can alienate potential community members. 

SPARK-32004 removed references to slave

 
Here is a IETF draft to fix up some of the most egregious examples
(master/slave, whitelist/backlist) with proposed alternatives.
https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1
Bug: Failed to serialize large MapStatuses
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/20 13:45
Affects Version/s: 2.3.4

Driver side exception:
{noformat}
20/07/07 02:22:26,366 ERROR [map-output-dispatcher-3] spark.MapOutputTrackerMaster:91 :
java.lang.NegativeArraySizeException
        at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
        at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:984)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply$mcV$sp(MapOutputTracker.scala:228)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
        at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:222)
        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:493)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
20/07/07 02:22:26,366 ERROR [map-output-dispatcher-5] spark.MapOutputTrackerMaster:91 :
java.lang.NegativeArraySizeException
        at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
        at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:984)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply$mcV$sp(MapOutputTracker.scala:228)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
        at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:222)
        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:493)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
20/07/07 02:22:26,366 ERROR [map-output-dispatcher-2] spark.MapOutputTrackerMaster:91 :
{noformat}
Bug: pyspark crashes in cluster mode with kafka structured streaming
Status: Open | Priority: Major | Resolution: 
Created: 23/Jul/20 15:15
Affects Version/s: 3.0.0

Hello,

{{I have been trying to run a pyspark script on Spark on Kubernetes and I have this error that crashed the application:}}

{{java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD)}}

 

I followed those steps:
 * for spark on kubernetes: [https://spark.apache.org/docs/latest/running-on-kubernetes.html] (that include building the image using docker-image-tool.sh on mac with -p flag)
 * Tried to use the image by the dev on GoogleCloudPlatform/spark-on-k8s-operator (gcr.io/spark-operator/spark-py:v3.0.0) and have the same issue
 * for kafka streaming: [https://spark.apache.org/docs/3.0.0/structured-streaming-kafka-integration.html#deploying]
 * {{When running the script manually in a jupyter notebook (jupyter/pyspark-notebook:latest, version 3.0.0) in local mode (with PYSPARK_SUBMIT_ARGS=--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0 pyspark-shell) it ran without issue}}
 * the command ran from the laptop is:

spark-submit --master k8s://[https://127.0.0.1:53979|https://127.0.0.1:53979/] --name spark-pi --deploy-mode cluster --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0 --conf spark.kubernetes.container.image=fifoosab/pytest:3.0.0.dev0 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.executor.request.cores=1 --conf spark.kubernetes.driver.request.cores=1 --conf spark.kubernetes.container.image.pullPolicy=Always local:///usr/bin/spark.py

 

{{full logs on the error in the attachements}}
Bug: Avoid encoding URL twice on https redirect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jul/20 14:31
Affects Version/s: 3.0.1

Currently, on https redirect, the original URL is encoded as an HTTPS URL. However, the original URL could be encoded already, so that the return result of method
UriInfo.getQueryParameters will contain encoded keys and values. For example, a parameter
order[0][dir] will become order%255B0%255D%255Bcolumn%255D after encoded twice, and the decoded
key in the result of UriInfo.getQueryParameters will be order%5B0%5D%5Bcolumn%5D.

To fix the problem, we try decoding the query parameters before encoding it. This is to make sure we encode the URL exactly once.
Bug: COALESCE should not reduce the child parallelism if it is Join
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Jul/20 08:18
Affects Version/s: 3.1.0

How to reproduce this issue:
{code:scala}
spark.range(100).createTempView("t1")
spark.range(200).createTempView("t2")
spark.sql("set spark.sql.autoBroadcastJoinThreshold=0")
spark.sql("select /*+ COALESCE(1) */ t1.* from t1 join t2 on (t1.id = t2.id)").show
{code}

The dag is:
 !COALESCE.png! 

A real case:
 !coalesce.png! 
 !repartition.png!
Bug: Use ExpressionEncoder for the return type of ScalaUDF to convert to catalyst type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/20 08:51
Affects Version/s: 3.1.0

Users now could register a UDF with Instant/LocalDate as return type even with 

spark.sql.datetime.java8API.enabled=false. However, the UDF can only be really used with spark.sql.datetime.java8API.enabled=true. This could make users confused.

The problem is we use ExpressionEncoder to ser/deser types when registering the UDF, but use Catalyst converters to ser/deser types, which is under control of  spark.sql.datetime.java8API.enabled,  when executing UDF.

If we could also use ExpressionEncoder to ser/deser types, similar to what we do for input parameter types, the, UDF could support Instant/LocalDate, event other combined complex types as well.
Bug: Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/20 03:01
Affects Version/s: 3.1.0

The EnsureRquirement.reorderJoinKeys can be improved to handle the following scenarios:
# If the keys cannot be reordered to match the left-side HashPartitioning, consider the right-side HashPartitioning.
# Handle PartitioningCollection, which may contain HashPartitioning
Bug: Expose end point on Master so that it can be informed about decommissioned workers out of band
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/20 23:16
Affects Version/s: 3.1.0

The use case here is to allow some external entity that has made a decommissioning decision to inform the Master (in case of Standalone scheduling mode)

The current decommissioning is triggered by the Worker getting getting a SIGPWR
 (out of band possibly by some cleanup hook), which then informs the Master
 about it. This approach may not be feasible in some environments that cannot
 trigger a clean up hook on the Worker.

Add a new post endpoint {{/workers/kill}} on the MasterWebUI that allows an
 external agent to inform the master about all the nodes being decommissioned in
 bulk. The workers are identified by either their {{host:port}} or just the host
 – in which case all workers on the host would be decommissioned.

This API is merely a new entry point into the existing decommissioning
 logic. It does not change how the decommissioning request is handled in
 its core.

The path /workers/kill is so chosen to be consistent with the other endpoint names on the MasterWebUI. 

Since this is a sensitive operation, this API will be disabled by default.
Bug: Keep advanced statistics when pruning partitions
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Jul/20 13:18
Affects Version/s: 3.1.0

The rule {{PruneFileSourcePartitions}} deletes the advances statistics (cardinality, column stats) of the underlying {{LogicalRelation}} and keeps only {{sizeInBytes}} as that is the only value we can be sure that is accurate.
I think we should keep all statistics as they are a good upper limit estimates and by keeping them other rules that depend on the presence of these statistics (like {{CostBasedJoinReorder}}) can still ran.
Bug: Avoid possible errors due to incorrect file size or type supplied in spark conf.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 07:21
Affects Version/s: 3.1.0

This would avoid failures, in case the files are a bit large or a user places a binary file inside the SPARK_CONF_DIR.

Both of which are not supported at the moment.

The reason is, underlying etcd store does limit the size of each entry to only 1.5 MiB.

[https://etcd.io/docs/v3.4.0/dev-guide/limit/] 

We can apply a straightforward approach of skipping files that cannot be accommodated within 1.5MiB limit (limit is configurable as per above link) and WARNING the user about the same.

For most use cases, this limit is more than sufficient, however a user may accidentally place a larger file and observe an unpredictable result or failures at run time.
Bug: Flaky Test: ExecutorAllocationManagerSuite.add executors default profile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/20 02:36
Affects Version/s: 3.1.0

 This test becomes flaky in Github Actions, see: https://github.com/apache/spark/pull/29072/checks?check_run_id=861689509
{code:java}
[info] - add executors default profile *** FAILED *** (33 milliseconds)
[info]   4 did not equal 2 (ExecutorAllocationManagerSuite.scala:132)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
[info]   at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503)
[info]   at org.apache.spark.ExecutorAllocationManagerSuite.$anonfun$new$7(ExecutorAllocationManagerSuite.scala:132)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:157)
[info]   at org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
[info]   at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:286)
[info]   at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
[info]   at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:59)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:59)
[info]   at org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
[info]   ...
{code}
Bug: Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/20 20:12
Affects Version/s: 3.0.0

We'll need to update to scalatest 3.2.0 in order to pick up the fix here, which fixes an incompatibility with Scala 2.13.3:

https://github.com/scalatest/scalatest/commit/7c89416aa9f3e7f2730a343ad6d3bdcff65809de

That's a big change unfortunately - 3.1 / 3.2 reorganized many classes. Fortunately it's just like import updates in 100 files.
Bug: Omit USING in CREATE TABLE via JDBC Table Catalog
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Jul/20 11:12
Affects Version/s: 3.1.0

Support creating tables in JDBC Table Catalog without USING, for instance:
{code:sql}
CREATE TABLE h2.test.new_table(i INT, j STRING)
{code}
Bug: Support adding a user provided config map.
Status: Open | Priority: Major | Resolution: 
Created: 08/Jul/20 07:25
Affects Version/s: 3.1.0

One of the challenge with this is, spark.properties is not user provided and is calculated based on certain factors. So a user provided config map, cannot be used as is to mount as SPARK_CONF_DIR, so it will have to be somehow augmented with the correct spark.properties.

Q, Do we support update to config map properties for an already running job?
Ans: No, since the spark.properties is calculated at the time of job submission, it cannot be updated on the fly and it is not supported by Spark at the moment for all the configuration values.

Q. What are the usecases, where supplying SPARK_CONF_DIR via a config map helps?
One of the use case, I can think of is programmatically submitting a `spark on k8s` job - e.g. spark as a service on a cloud deployment may find this feature useful.
Bug: Add basic partition command for datasourcev2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 15:57
Affects Version/s: 3.1.0

This Jira is trying to add basic partition command API, `AlterTableAddPartitionExec` and `AlterTableDropPartitionExec`,  to support operating datasourcev2 partitions. This will use the new partition API defined in [SPARK-31694|https://issues.apache.org/jira/browse/SPARK-31694].
Bug: Add K8s IT for conf propagation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 07:23
Affects Version/s: 3.1.0

An integration test by placing a configuration file in SPARK_CONF_DIR, and verifying it is loaded on the executors in both client and cluster deploy mode. 
For this, a log4j.properties file is a good candidate for testing.
Bug: Inconsistent NULL conversions to strings 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/20 18:22
Affects Version/s: 3.1.0

1. It is impossible to distinguish empty string and null, for instance:
{code:scala}
scala> Seq(Seq(""), Seq(null)).toDF().show
+-----+
|value|
+-----+
| []|
| []|
+-----+
{code}
2. Inconsistent NULL conversions for top-level values and nested columns, for instance:
{code:scala}
scala> sql("select named_struct('c', null), null").show
+---------------------+----+
|named_struct(c, NULL)|NULL|
+---------------------+----+
| []|null|
+---------------------+----+
{code}
3. `.show()` is different from conversions to Hive strings, and as a consequence its output is different from `spark-sql` (sql tests):
{code:sql}
spark-sql> select named_struct('c', null) as struct;
{"c":null}
{code}
{code:scala}
scala> sql("select named_struct('c', null) as struct").show
+------+
|struct|
+------+
| []|
+------+
{code}
Bug: Application entry parsing fails because DriverWrapper registered instead of the normal driver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 12:22
Affects Version/s: 3.1.0

In some cases DriverWrapper registered by DriverRegistry which causes exception in PostgresConnectionProvider:
https://github.com/apache/spark/blob/371b35d2e0ab08ebd853147c6673de3adfad0553/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/DriverRegistry.scala#L53
Bug: Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/20 07:13
Affects Version/s: 3.1.0

In Scala 2.13, this causes several UT failures because `HighlyCompressedMapStatus.readExternal` doesn't initialize this field. The following two are the examples.
- org.apache.spark.rdd.RDDSuite
{code}
  Cause: java.lang.NoSuchFieldError: numNonEmptyBlocks
  at org.apache.spark.scheduler.HighlyCompressedMapStatus.<init>(MapStatus.scala:181)
  at org.apache.spark.scheduler.HighlyCompressedMapStatus$.apply(MapStatus.scala:280)
  at org.apache.spark.scheduler.MapStatus$.apply(MapStatus.scala:73)
  at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:231)
  at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:179)
  at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
  at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
  at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
  at org.apache.spark.scheduler.Task.run(Task.scala:127)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:464)
{code}

- org.apache.spark.scheduler.MapStatusSuite
{code}
*** RUN ABORTED ***
  java.lang.NoSuchFieldError: numNonEmptyBlocks
  at org.apache.spark.scheduler.HighlyCompressedMapStatus.<init>(MapStatus.scala:181)
  at org.apache.spark.scheduler.HighlyCompressedMapStatus$.apply(MapStatus.scala:280)
{code}
Bug: Implement ALTER TABLE in JDBC Table Catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 06:13
Affects Version/s: 3.1.0

The PR https://github.com/apache/spark/pull/29168 adds basic implementation of JDBC Table Catalog. This ticket aims to support table altering in:
- JDBC dialects
- JDBCTableCatalog
and to add tests.
Bug: Kafka events are missing magic byte
Status: Open | Priority: Major | Resolution: 
Created: 17/Jul/20 04:35
Affects Version/s: 3.0.0

Please refer to the documentation link for to_avro and from_avro.[ http://spark.apache.org/docs/latest/sql-data-sources-avro.html|http://spark.apache.org/docs/latest/sql-data-sources-avro.html]

Tested the to_avro function by making sure that data is sent to Kafka topic. But when a Confluent Avro consumer is used to read data from the same topic, the consumer fails with an error message that event is missing the magic byte. 

Used another topic to simulate reads from Kafka and further deserialization using from_avro. Use case is, use a Confluent Avro producer to produce a few events. And when I attempt to read this topic using structured streaming and applying the function from_avro, it fails with a message indicating that malformed records are present. 

Using from_avro (deserialization) and to_avro (serialization) from Spark, only works with Spark. And other consumers outside of Spark which do not use this approach are failing.
Bug: Add dropFields method to Column class
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/20 15:43
Affects Version/s: 3.0.0

Based on the discussions in the parent ticket (SPARK-22231), add a new {{dropFields}} method to the {{Column}} class. 
This method should allow users to drop a column nested inside a StructType Column (with similar semantics to the existing {{drop}} method on {{Dataset}}).
It should also be able to handle deeply nested columns through the same API. 
This is similar to the {{withField}} method that was recently added in SPARK-31317 and likely we can re-use some of that "infrastructure."

The public-facing method signature should be something along the following lines: 

{noformat}
def dropFields(fieldNames: String*): Column
{noformat}