av9ash/GitBugs huggingface dataset:
Bug: Add Row to __all__ in pyspark.sql.types
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Oct/21 15:05
Affects Version/s: 3.3.0

Currently {{Row}}, defined in {{pyspark.sql.types}} is exported from {{pyspark.sql}} but not types. It means that {{from pyspark.sql.types import *}} won't import {{Row}}.

It might be counter-intuitive, especially when we import {{Row}} from {{types}} in {{examples}}.

Should we add it to {{__all__}}?
Bug: What ciphers spark support for internode communication?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 12/Oct/21 14:10
Affects Version/s: 3.1.2

{{Spark documentation mentions this:}}
 {{[https://spark.apache.org/docs/3.0.0/security.html]}}
{code:java}
spark.network.crypto.config.*
"Configuration values for the commons-crypto library, such as which cipher implementations to use. The config name should be the name of commons-crypto configuration without the commons.crypto prefix."{code}
{{What this means?}}
 {{If I leave it to None what will happen? There won't be any encryption used or will it fallback to some default one?}}
 {{The common-crypto mentions that it uses JCE or OPENSSL implementations, but says nothing about the ciphers.}}
 {{Does it support everything the given JVM does?}}

{{The documentation is vague on this.}}

{{However the spark ui part for the security is clear:}}
{code:java}
${ns}.enabledAlgorithms
A comma-separated list of ciphers. The specified ciphers must be supported by JVM. The reference list of protocols can be found in the "JSSE Cipher Suite Names" section of the Java security guide. The list for Java 8 can be found at this page. Note: If not set, the default cipher suite for the JRE will be used.{code}
{{ }}
 {{So what will happen if I leave spark.network.crypto.config.* to None?}}
 {{And what ciphers are supported?}}
Bug: Future typing errors in pyspark.pandas
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Oct/21 10:13
Affects Version/s: 3.3.0

The following problems are detected on master with mypy 0.920
{code:java}
$ git rev-parse HEAD             
36b3bbc0aa9f9c39677960cd93f32988c7d7aaca
$ mypy --version                 
mypy 0.920+dev.332b712df848cd242987864b38bd237364654532
$ mypy --config-file mypy.ini pyspark
pyspark/pandas/indexes/base.py:184: error: Incompatible types in assignment (expression has type "CategoricalIndex", variable has type "MultiIndex")  [assignment]
pyspark/pandas/indexes/base.py:188: error: Incompatible types in assignment (expression has type "Int64Index", variable has type "MultiIndex")  [assignment]
pyspark/pandas/indexes/base.py:192: error: Incompatible types in assignment (expression has type "Float64Index", variable has type "MultiIndex")  [assignment]
pyspark/pandas/indexes/base.py:197: error: Incompatible types in assignment (expression has type "DatetimeIndex", variable has type "MultiIndex")  [assignment]
pyspark/pandas/indexes/base.py:199: error: Incompatible types in assignment (expression has type "Index", variable has type "MultiIndex")  [assignment]
pyspark/pandas/indexes/base.py:201: error: "MultiIndex" has no attribute "_anchor"  [attr-defined]
{code}
Bug: Make BadRecordException serializable
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Oct/21 03:14
Affects Version/s: 3.2.0

Migrating a Spark application from 2.4.x to 3.1.x and finding a difference in the exception chaining behavior. In a case of parsing a malformed CSV, where the root cause exception should be {{Caused by: java.lang.RuntimeException: Malformed CSV record}}, only the top level exception is kept, and all lower level exceptions and root cause are lost. Thus, when we call {{ExceptionUtils.getRootCause}} on the exception, we still get itself.
The reason for the difference is that {{RuntimeException}} is wrapped in {{BadRecordException}}, which has unserializable fields. When we try to serialize the exception from tasks and deserialize from scheduler, the exception is lost.
This PR makes unserializable fields of {{BadRecordException}} transient, so the rest of the exception could be serialized and deserialized properly.
Bug: Overriding parameters
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 04/Oct/21 10:54
Affects Version/s: 3.1.1

Hello team, One quick question.

I read somewhere in the doc - Order of priority

Value set up in spark-submit overrides value set up in code overrides value set up in .conf

Is this correct understanding? If so, we may have problem.

I was trying to set spark.kubernetes.namespace. I had used default in code but wanted to override in submit-submit. Application kept failing. Finally I had to remove from code to make it work.

Same for some other kubernetes setups like PersistentVolume set up.
Bug: Maven build failed in windows!
Status: Resolved | Priority: Minor | Resolution: Done
Created: 17/Oct/21 14:17
Affects Version/s: 3.2.0

I pulled the latest Spark master code on my local windows 10 computer and executed the following command:
{code:java}
mvn -DskipTests clean install{code}
Build failed!

!image-2021-10-17-22-18-16-616.png!
{code:java}
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project spark-core_2.12: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\bigdata\spark\core"): CreateProcess error=2{code}
It seems that the plugin: maven-antrun-plugin cannot run because of windows no bash. 

The following code comes from pom.xml in spark-core module.
{code:java}
<plugin>

  <groupId>org.apache.maven.plugins</groupId>

  <artifactId>maven-antrun-plugin</artifactId>

  <executions>

    <execution>

      <phase>generate-resources</phase>

      <configuration>

        <!-- Execute the shell script to generate the spark build information. -->

        <target>

          <exec executable="bash">

            <arg value="${project.basedir}/../build/spark-build-info"/>

            <arg value="${project.build.directory}/extra-resources"/>

            <arg value="${project.version}"/>

          </exec>

        </target>

      </configuration>

      <goals>

        <goal>run</goal>

      </goals>

    </execution>

  </executions>

</plugin>

{code}
Bug: Confusing tgz filename for download
Status: Open | Priority: Minor | Resolution: 
Created: 20/Oct/21 04:59
Affects Version/s: 3.2.0

In the Spark download webpage [https://spark.apache.org/downloads.html], the package type dropdown says "Hadoop 3.3", but the Download Spark tgz filename contains "hadoop3.2" in it.  It is confusing; which version is correct?

 

Download Apache Spark(TM)
 # Choose a Spark release: 3.2.0 (Oct 13 2021)
 # Choose a package type: Pre-built for Apache Hadoop 3.3 and later
 # Download Spark: spark-3.2.0-bin-hadoop3.2.tgz
Bug: Fix the R test of FPGrowthModel for Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Oct/21 14:12
Affects Version/s: 3.3.0

Similar to the issue filed in SPARK-37059, the R test of FPGrowthModel assumes that the result records returned by FPGrowthModel.freqItemsets are sorted by a certain kind of order but it's wrong.
As a result, the test fails with Scala 2.13.

{code}
 ══ Failed ══════════════════════════════════════════════════════════════════════
── 1. Failure (test_mllib_fpm.R:42:3): spark.fpGrowth ──────────────────────────
`expected_itemsets` not equivalent to `itemsets`.
Component “items”: Component 1: Component 1: 1 string mismatch
Component “items”: Component 2: Length mismatch: comparison on first 1 components
Component “items”: Component 2: Component 1: 1 string mismatch
Component “items”: Component 3: Length mismatch: comparison on first 1 components
Component “items”: Component 4: Length mismatch: comparison on first 1 components
Component “items”: Component 4: Component 1: 1 string mismatch
Component “items”: Component 5: Length mismatch: comparison on first 1 components
Component “items”: Component 5: Component 1: 1 string mismatch
Component “freq”: Mean relative difference: 0.5454545
{code}
Bug: why spark-X.X.X-bin-without-hadoop.tgz does not provide spark-hive_X.jar (and spark-hive-thriftserver_X.jar)
Status: Open | Priority: Minor | Resolution: 
Created: 27/Oct/21 12:42
Affects Version/s: 3.1.2

Hi,

As my deployment is having its own Hadoop(+Hive) installed, I have tried to install Spark using  its bundle without Hadoop. I suspect that some jars are missing that are present in the corresponding spark-X.X.X-bin-hadoop3.2.tgz.

After comparing their contents both spark-hive_2.12-X.X.X.jar and spark-hive-thriftserver_2.12-X.X.X.jar are not in the spark-X.X.X-bin-without---hadoop.tgz. And I don't know if some others should also be there.

Thanks,

Patrice
Bug: Add RewriteLateralSubquery rule into nonExcludableRules
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Oct/21 02:42
Affects Version/s: 3.3.0

Lateral Join has no meaning without rule `RewriteLateralSubquery`. So now if we set `spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.RewriteLateralSubquery`, the lateral join query will fail with:
{code:java}
java.lang.AssertionError: assertion failed: No plan for LateralJoin lateral-subquery#218
{code}
Bug: `select format_string('%0$s', 'Hello')` has different behavior when using java 8 and Java 17
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Oct/21 03:44
Affects Version/s: 3.3.0

{code:java}
--PostgreSQL throw ERROR:  format specifies argument 0, but arguments are numbered from 1
select format_string('%0$s', 'Hello');
{code}
Execute with Java 8
{code:java}
-- !query
select format_string('%0$s', 'Hello')
-- !query schema
struct<format_string(%0$s, Hello):string>
-- !query output
Hello
{code}
Execute with Java 17
{code:java}
-- !query
select format_string('%0$s', 'Hello')
-- !query schema
struct<>
-- !query output
java.util.IllegalFormatArgumentIndexException
Illegal format argument index = 0
{code}
Bug: Missing overloads in functions accepting both varargs and single arg collection
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Oct/21 08:22
Affects Version/s: 3.1.0

There are a few functions that have vararg signature, but can accept a single item ({{list}} or {{tuple}}):
 - {{def create_map(*cols: "ColumnOrName") -> Column: ...}}
 - {{def struct(*cols: "ColumnOrName") -> Column: ...}}
 - {{def array(*cols: "ColumnOrName") -> Column: ...}}
 - {{def map_concat(*cols: "ColumnOrName") -> Column: ...}}

 

We should add overloads of the following form

 
{code:python}
@overload
def _(cols: Union[List["ColumnOrName"], Tuple["ColumnOrName", ...]]) -> Column:
    ...

@overload
def _(*cols: "ColumnOrName") -> Column:
    ...

def _(
    *cols: Union["ColumnOrName", List["ColumnOrName"], Tuple["ColumnOrName", ...]]
) -> Column:
   ...
{code}
Bug: Improve ORC RecordReader's error message
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Oct/21 03:21
Affects Version/s: 3.2.0

In below error message, we don't know the actual file path
{code}

21/10/19 18:12:58 ERROR Executor: Exception in task 34.1 in stage 14.0 (TID 257)
java.lang.ArrayIndexOutOfBoundsException: 1024
	at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:292)
	at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.nextVector(TreeReaderFactory.java:1820)
	at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.nextVector(TreeReaderFactory.java:1517)
	at org.apache.orc.impl.ConvertTreeReaderFactory$DateFromStringGroupTreeReader.nextVector(ConvertTreeReaderFactory.java:1802)
	at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:2059)
	at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1324)
	at org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.nextBatch(OrcColumnarBatchReader.java:196)
	at org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.nextKeyValue(OrcColumnarBatchReader.java:99)
	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:503)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:132)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}
Bug: Addition of forany and forall semantics to Spark Dataframes
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 31/Oct/21 03:38
Affects Version/s: 3.2.0

Spark dataframes currently do not have semantics of forany and forall.

Scala collections do have these semantics.

It would be nice to have them as an API as there is potential for implementing them in unoptimized form.

 

E.g. forany might be implemented as df.filter(<condition>).count > 0, which is not optimal
Bug: Missing dependencies for hadoop-azure
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Oct/21 16:18
Affects Version/s: 3.2.0

If we build spark distribution (from 3.2.0 release tag) via
{code:java}
./dev/make-distribution.sh --name custom-spark --tgz  -Pkubernetes -Phadoop-cloud{code}
and try to access Azure Data Lake storage like:
{code:java}
sqlContext.read.parquet("my_data"){code}
{{then read operation fails with error:}}
{code:java}
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper{code}
{{And indeed I can see exclusion being applied in multiple places for org.codehaus.jackson:jackson-mapper-asl.}}

{{Surprisingly 3.1.2 is fine, but I believe it is because avro version was changed for 3.2.0 and update stopped bringing in such dependency.}}

{{Shall the build allow to bring extra dependencies in?}}
Bug: Supplement the missing Java 11 benchmark result files
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Oct/21 08:44
Affects Version/s: 3.3.0

CharVarcharBenchmark-results.txt and UpdateFieldsBenchmark-results.txt exist in the project, but CharVarcharBenchmark-jdk11-results.txt and UpdateFieldsBenchmark-jdk11-results.txt are missing
Bug: OpenHashMap should be serializable without reference tracking
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Oct/21 10:12
Affects Version/s: 3.2.0

The current implementation of OpenHashMap does not serialize without kryo reference tracking turned on. This is unexpected from a simple type like OpenHashMap, and forces the users to turn of reference tracking to use code where OpenHashMap is used.
Bug: Support Submitting Jobs to Cloud Providers Managed Spark Clusters
Status: Open | Priority: Minor | Resolution: 
Created: 26/Oct/21 01:36
Affects Version/s: 3.2.0

To be able to submit jobs to prominent cloud providers managed spark clusters, "spark-submit" can be enhanced. For example, to submit job to "google cloud dataproc", the "spark-submit" can be enhanced to issue "gcloud dataproc jobs submit spark ..." when "–master gcd://cluster-name" arg is used. Once this feature is accepted and prioritized, then it can be rolled out in current and future versions of spark and also back ported to a few previous versions. I can raise the pull request.
Bug: Pandas groupby UDFs would benefit from automatically redistributing data on the groupby key in order to prevent network issues running udf
Status: Open | Priority: Minor | Resolution: 
Created: 22/Oct/21 20:28
Affects Version/s: 3.1.2

when running high cardinality pandas udf groupby steps (100,000s+ of unique groups) - jobs will either fail or have high amount of task failures due to network errors on larger clusters 100+ nodes - this was not the specific code causing issues but should be close to representative:


from pyspark.sql.functions import pandas_udf, PandasUDFType
from pyspark.sql.functions import rand
from fancyimpute import IterativeSVD
import numpy as np
import pandas as pd
​
df = spark.range(0, 100000).withColumn('v', rand())
@pandas_udf(df.schema, PandasUDFType.GROUPED_MAP)
def solver(pdf):
pd.DataFrame(data=IterativeSVD(verbose=False).fit_transform(pdf.to_numpy()))
return pdf
​
df.groupby('id').apply(solver).count()
 
df.repartition('id') – this is required to fix it - can we make this automatically happen without any adverse impacts?
Bug: Custom V2 Metrics uses wrong classname for lookup
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Oct/21 19:39
Affects Version/s: 3.2.0

Currently CustomMetrics uses `getCanonicalName` to get the metric type name

https://github.com/apache/spark/blob/38493401d18d42a6cb176bf515536af97ba1338b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/CustomMetrics.scala#L31-L33

But when using reflection we need to use the original type name.

Here is an example when working with an inner class

{code:java title="CanonicalName vs Name"}
Class.getName = org.apache.iceberg.spark.source.SparkBatchScan$FilesScannedMetric
Class.getCanonicalName =
org.apache.iceberg.spark.source.SparkBatchScan.FilesScannedMetric
{code}


The "$" name is required to look up this class while the "." version will fail with CNF.
Bug: Ensure the sort order of the output in the PySpark doctests
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Oct/21 12:46
Affects Version/s: 3.3.0

The collect_set builtin function doesn't ensure the sort order of its result for each row. FPGrouthModel.freqItemsets also doesn't ensure the sort order of the result rows.
Nevertheless, their PySpark doctests assume a certain kind of sort order, causing that such doctests fail with Scala 2.13.
Bug: Fix wrong DocSearch facet filter in release-tag.sh
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Oct/21 06:52
Affects Version/s: 3.2.0

In release-tag.sh, the DocSearch facet filter should be updated as the release version before creating  git tag. 
If we missed the step, the facet filter is wrong in the new release doc:  https://github.com/apache/spark/blame/v3.2.0/docs/_config.yml#L42
Bug: Extend python test runner by logging out the temp output files
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Oct/21 08:36
Affects Version/s: 3.3.0

I was running a python test which was extremely slow and I was surprised the unit-tests.log has not been even created. Looked into the code and as I got the tests can be executed in parallel and each one has its own temporary output file which is only added to the unit-tests.log when a test is finished with a failure (after acquiring a lock to avoid parallel write on unit-tests.log). 

To avoid such a confusion it would make sense to log out the path of those temporary output files.
Bug: WholeStageCodegenSparkSubmitSuite Failed with Java 17 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Oct/21 11:35
Affects Version/s: 3.3.0

WholeStageCodegenSparkSubmitSuite test failed when use Java 17
{code:java}
 2021-10-14 04:32:38.038 - stderr> Exception in thread "main" org.scalatest.exceptions.TestFailedException: 16 was not greater than 16
  2021-10-14 04:32:38.038 - stderr> 	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
  2021-10-14 04:32:38.038 - stderr> 	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
  2021-10-14 04:32:38.038 - stderr> 	at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
  2021-10-14 04:32:38.038 - stderr> 	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite$.main(WholeStageCodegenSparkSubmitSuite.scala:82)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.sql.execution.WholeStageCodegenSparkSubmitSuite.main(WholeStageCodegenSparkSubmitSuite.scala)
  2021-10-14 04:32:38.038 - stderr> 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2021-10-14 04:32:38.038 - stderr> 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  2021-10-14 04:32:38.038 - stderr> 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2021-10-14 04:32:38.038 - stderr> 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
  2021-10-14 04:32:38.038 - stderr> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
Bug: Pin actions to a full length commit SHA
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Oct/21 19:25
Affects Version/s: 3.1.2

Pinning an action to a full-length commit SHA is currently the only way to use an action as
an immutable release. Pinning to a particular SHA helps mitigate the risk of a bad actor adding a
backdoor to the action's repository, as they would need to generate a SHA-1 collision for
a valid Git object payload.

[https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions]

[https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies]
Bug: Add checks to DebugFilesystem to ensure that FS operations are not performed in the DAGScheduler event loop
Status: Open | Priority: Minor | Resolution: 
Created: 14/Oct/21 21:30
Affects Version/s: 3.0.0

As [~yuchen.huo] suggested at [https://github.com/apache/spark/pull/34265#discussion_r728805893,] we explore modifying {{DebugFilesystem}} to throw exceptions in case filesystem operations are performed from inside of the DAGScheduler's event processing thread. This could help prevent future issues that are similar to SPARK-23626.