av9ash/GitBugs huggingface dataset:
Bug: Make SocketAuthServer socket timeout configurable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/20 08:20
Affects Version/s: 2.4.7

In SPARK-21551 the socket timeout for the Pyspark applications has been increased from 3 to 15 seconds. However it is still hardcoded.
In certain situations even the 15 seconds is not enough, so it should be made configurable. 
This is requested after seeing it in real-life workload failures.
Also it has been suggested and requested in an earlier comment in [SPARK-18649|https://issues.apache.org/jira/browse/SPARK-18649?focusedCommentId=16493498&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16493498]

In 
Spark 2.4 it is under
[PythonRDD.scala|https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala#L899]
in Spark 3.x the code has been moved to
[SocketAuthServer.scala|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/security/SocketAuthServer.scala#L51]
{code}
serverSocket.setSoTimeout(15000)
{code}
Please include this in both 2.4 and 3.x branches.
Bug: Spark 3: Partition count changing on dataframe select cols
Status: Open | Priority: Major | Resolution: 
Created: 15/Oct/20 15:49
Affects Version/s: 3.0.0

I am noticing a difference in behaviour on upgrading to spark 3 where the NumPartitions are changing on df.select which causing my zip operations to fail on mismatch. With spark 2.4.4 it works fine. This does not happen with filter but only with select cols
{code:java}
spark = SparkSession.builder.appName("pytest-pyspark-local-testing"). \ master("local[5]"). \ config("spark.executor.memory", "2g"). \ config("spark.driver.memory", "2g"). \ config("spark.ui.showConsoleProgress", "false"). \ config("spark.sql.shuffle.partitions",10). \ config("spark.sql.optimizer.dynamicPartitionPruning.enabled","false").getOrCreate()
{code}
 

With Spark 2.4.4:
  df = spark.table("tableA")
 print(df.rdd.getNumPartitions()) #10
 new_df = df.filter("id is not null")
 print(new_df.rdd.getNumPartitions()) #10
 new_2_df = df.select("id")
 print(new_2_df.rdd.getNumPartitions()) #10
  

With Spark 3.0.0:
 df = spark.table("tableA")
 print(df.rdd.getNumPartitions()) #10
 new_df = df.filter("id is not null")
 print(new_df.rdd.getNumPartitions()) #10

new_1_df = df.select("*")
 print(new_1_df.rdd.getNumPartitions()) #10


 new_2_df = df.select("id")
 print(new_2_df.rdd.getNumPartitions()) #1
 See the last line where it changes to 1 partition from initial 10. Any thoughts?
Bug: Remove dependencies(scalatest,scalactic) from Benchmark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/20 00:55
Affects Version/s: 3.0.1

This ticket aims at removing `assert` from `Benchmark` for making it easier to run benchmark codes via `spark-submit`.
Since the current `Benchmark` (`master` and `branch-3.0`) has `assert`, we need to pass the proper jars of `scalatest` and `scalactic`;
 - scalatest-core_2.12-3.2.0.jar
 - scalatest-compatible-3.2.0.jar
 - scalactic_2.12-3.0.jar
{code}
./bin/spark-submit --jars scalatest-core_2.12-3.2.0.jar,scalatest-compatible-3.2.0.jar,scalactic_2.12-3.0.jar,./sql/catalyst/target/spark-catalyst_2.12-3.1.0-SNAPSHOT-tests.jar,./core/target/spark-core_2.12-3.1.0-SNAPSHOT-tests.jar --class org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark ./sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar --data-location /tmp/tpcds-sf1
{code}
Bug: Check the metadata key 'org.apache.spark.legacyDateTime' in Avro/Parquet files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/20 18:57
Affects Version/s: 3.1.0

Add tests to check that the metadata key 'org.apache.spark.legacyDateTime' is saved correctly depending on the SQL configs:
# spark.sql.legacy.avro.datetimeRebaseModeInWrite
# spark.sql.legacy.parquet.datetimeRebaseModeInWrite
Bug: Replace the existing Maven job for Scala 2.13 in Github Actions with SBT job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/20 21:58
Affects Version/s: 3.1.0

SPARK-32926 added a build test to GitHub Action for Scala 2.13 but it's only with Maven.
As SPARK-32873 reported, some compilation error happens only with SBT so I think we need to add another build test to GitHub Action for SBT.
Bug: Incorrect nested complex JSON fields raise an exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 13:07
Affects Version/s: 3.0.2

The code below:
{code:scala}
val pokerhand_raw = Seq("""[{"cards": [19], "playerId": 123456}]""").toDF("events")
    val event = new StructType()
      .add("playerId", LongType)
      .add("cards", ArrayType(
        new StructType()
          .add("id", LongType)
          .add("rank", StringType)))
    val pokerhand_events = pokerhand_raw
      .select(explode(from_json($"events", ArrayType(event))).as("event"))
    pokerhand_events.show
{code}
throw the exception in the PERMISSIVE mode (default):
{code:java}
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
  at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray(rows.scala:48)
  at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray$(rows.scala:48)
  at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:195)
  at org.apache.spark.sql.catalyst.expressions.JsonToStructs.$anonfun$converter$2(jsonExpressions.scala:560)
  at org.apache.spark.sql.catalyst.expressions.JsonToStructs.nullSafeEval(jsonExpressions.scala:597)
  at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:461)
  at org.apache.spark.sql.catalyst.expressions.ExplodeBase.eval(generators.scala:313)
  at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$8(GenerateExec.scala:108)
{code}
The same works in Spark 2.4:
{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.6
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_265)
...
scala> pokerhand_events.show()
+-----+
|event|
+-----+
+-----+
{code}
Bug: ScalarSubquery should returns the first two rows to avoid Driver OOM 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/20 10:33
Affects Version/s: 3.1.0

{noformat}
Exception in thread "subquery-2871" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
 at scala.collection.mutable.ResizableArray$class.ensureSize(ResizableArray.scala:103)
 at scala.collection.mutable.ArrayBuffer.ensureSize(ArrayBuffer.scala:48)
 at scala.collection.mutable.ArrayBuffer.$plus$eq(ArrayBuffer.scala:84)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1$$anonfun$apply$2.apply(SparkPlan.scala:352)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1$$anonfun$apply$2.apply(SparkPlan.scala:352)
 at scala.collection.Iterator$class.foreach(Iterator.scala:893)
 at org.apache.spark.sql.execution.SparkPlan$$anon$1.foreach(SparkPlan.scala:330)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1.apply(SparkPlan.scala:352)
 at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeCollect$1.apply(SparkPlan.scala:351)
 at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
 at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
 at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:351)
 at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.executeCollect(AdaptiveSparkPlanExec.scala:274)
 at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1$$anonfun$apply$3.apply(basicPhysicalOperators.scala:830)
 at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1$$anonfun$apply$3.apply(basicPhysicalOperators.scala:827)
 at org.apache.spark.sql.execution.SQLExecution$$anonfun$withExecutionId$1.apply(SQLExecution.scala:132)
 at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:156)
 at org.apache.spark.sql.execution.SQLExecution$.withExecutionId(SQLExecution.scala:129)
 at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1.apply(basicPhysicalOperators.scala:827)
 at org.apache.spark.sql.execution.SubqueryExec$$anonfun$relationFuture$1.apply(basicPhysicalOperators.scala:827)
 at scala.
{noformat}
Bug: Deduplicate deterministic PythonUDF calls
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/20 14:11
Affects Version/s: 3.1.0

We run into an issue where a customer created a column with an expensive PythonUDF call and build a very complex logic on the the top of that column as new derived columns. Due to `CollapseProject` and `ExtractPythonUDFs` rules the UDF is called ~1000 times for each row which degraded the performance of the query significantly.

The `ExtractPythonUDFs` rule could deduplicate deterministic UDFs so as to avoid performance degradation.
Bug: unify temp view and permanent view behaviors
Status: Resolved | Priority: Major | Resolution: Done
Created: 14/Oct/20 01:58
Affects Version/s: 3.1.0

Currently, temp view store mapping of temp view name and its logicalPlan, and permanent view store in HMS stores its origin SQL text.

So for permanent view, when try to refer the permanent view, its SQL text will be parse-analyze-optimize-plan again with current SQLConf and SparkSession context, so it might keep changing when the SQLConf and context is different each time.

In order the unify the behaviors of temp view and permanent view, proposed that we keep its origin SQLText for both temp and permanent view, and also keep record of the SQLConf when the view was created. Each time we try to refer the view, we using the Snapshot SQLConf to parse-analyze-optimize-plan the SQLText, in this way, we can make sure the output of the created view to be stable.
Bug: avro format does not propagate Hadoop config from DS options to underlying HDFS file system
Status: Closed | Priority: Major | Resolution: Fixed
Created: 07/Oct/20 21:02
Affects Version/s: 3.1.0

When running:
{code:java}
spark.read.format("avro").options(conf).load(path)
{code}
The underlying file system will not receive the `conf` options.
Bug: Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/20 00:11
Affects Version/s: 2.4.7

Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.

E.g.,:
{code:java}
spark.range(0, 100000, 1, 1).write.parquet(path)

spark.conf.set("spark.sql.columnVector.offheap.enabled", True)

def f(x):
    return 0

fUdf = udf(f, LongType())

spark.read.parquet(path).select(fUdf('id')).head()
{code}
This is because, the Python evaluation consumes the parent iterator in a separate thread and it consumes more data from the parent even after the task ends and the parent is closed. If an off-heap column vector exists in the parent iterator, it could cause segmentation fault which crashes the executor.
Bug: Replace Probot Autolabeler with Github Action
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/20 07:12
Affects Version/s: 3.0.1

The Probot Autolabeler that we were using in both the Iceberg and the Spark repo is no longer working. I've confirmed that with the devleper, github user [at]mithro, who has indicated that the Probot Autolabeler is end of life and will not be maintained moving forward.


PRs have not been labeled for a few weeks now.

 

As I'm already interfacing with ASF Infra to have the probot permissions revoked from the Iceberg repo, and I've already submitted a patch to switch Iceberg to the standard github labeler action, I figured I would go ahead and volunteer myself to switch the Spark repo as well.

I will have a patch to switch to the new github labeler open within a few days.

 

Also thank you [~blue] (or [~holden]) for shepherding this! I didn't exactly ask, but it was understood in our group meeting for Iceberg that I'd be converting our labeler there so I figured I'd tackle the spark issue while I'm getting my hands into the labeling configs anyway =)
Bug: Spark 3 / Cats 2.2.0 classpath issue
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 06/Oct/20 19:03
Affects Version/s: 3.0.1

A small project with minimal dependencies as well as instructions on how to reproduce the issue is available at:

[https://github.com/samidalouche/spark3-cats220]

Executing this code works fine with cats 2.1.1 but fails with cats 2.2.0, which is quite surprising since the spark and cats dependencies are pretty much distinct from each other.

 
{code:java}
java.lang.NoSuchMethodError: 'void cats.kernel.CommutativeSemigroup.$init$(cats.kernel.CommutativeSemigroup)'
 at cats.UnorderedFoldable$$anon$1.<init>(UnorderedFoldable.scala:78)
 at cats.UnorderedFoldable$.<init>(UnorderedFoldable.scala:78)
 at cats.UnorderedFoldable$.<clinit>(UnorderedFoldable.scala)
 at cats.data.NonEmptyListInstances$$anon$2.<init>(NonEmptyList.scala:539)
 at cats.data.NonEmptyListInstances.<init>(NonEmptyList.scala:539)
 at cats.data.NonEmptyList$.<init>(NonEmptyList.scala:458)
 at cats.data.NonEmptyList$.<clinit>(NonEmptyList.scala)
 at catsspark.Boom$.assumeValid_$bang(boom.scala:19)
 at catsspark.Boom$.boom(boom.scala:14)
 ... 47 elided{code}
Thanks in advance for looking into this.


I submitted the same issue to cat's bug tracker: https://github.com/typelevel/cats/issues/3628
Bug: Spark 2.3 vs Spark 1.6 collect_list giving different schema
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/Oct/20 09:58
Affects Version/s: 2.3.4

Hi,

I am migrating from spark 1.6 to spark 2.3. However in collect_list I am getting different schema.

 
{code:java}
val df_date_agg = df
    .groupBy($"a",$"b",$"c")
    .agg(sum($"d").alias("data1"),sum($"e").alias("data2"))
    .groupBy($"a")
    .agg(collect_list(array($"b",$"c",$"data1")).alias("final_data1"),
         collect_list(array($"b",$"c",$"data2")).alias("final_data2"))
{code}
When I am running above line in spark 1.6 getting below schema

 

 
{code:java}
 |-- final_data1: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- final_data2: array (nullable = true)
 |    |-- element: string (containsNull = true)
{code}
 

 

but in spark 2.3 schema changed to 

 
{code:java}
|-- final_data1: array (nullable = true)
 |    |-- element: array (containsNull = true)
 |    |    |-- element: string (containsNull = true)
 |-- final_data1: array (nullable = true)
 |    |-- element: array (containsNull = true)
 |    |    |-- element: string (containsNull = true)
{code}
 

 

In Spark 1.6 array($"b",$"c",$"data1") is converting to string like this 
{code:java}
'[2020-09-26, Ayush, 103.67]'
{code}
In spark 2.3 it is converted to WrappedArray
{code:java}
WrappedArray(2020-09-26, Ayush, 103.67)
{code}
I want to keep my schema as it is Otherwise all the dependent codes have to change.

 

Thanks
Bug: Support Column inputs in PySpark ordering functions (asc*, desc*)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 08:47
Affects Version/s: 3.1.0

According to SPARK-26979, PySpark functions should support both {{Column}} and {{str}} arguments, when possible.

However, the following ordering support only {{str}}

- {{asc}}
- {{desc}}
- {{asc_nulls_first}}
- {{asc_nulls_last}}
- {{desc_nulls_first}}
- {{desc_nulls_last}}

support only {{str}}. This is because Scala side doesn't provide {{Column => Column}} variants.

To fix this, we do one of the following:

- Call corresponding {{Column}} methods as [suggested|https://github.com/apache/spark/pull/30143#discussion_r512366978] by  [~hyukjin.kwon]
- Add missing signatures on Scala side.
Bug: Improve error handling on Pandas to Arrow conversion failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Oct/20 06:38
Affects Version/s: 3.0.1

Currently, when converting from Pandas to Arrow for Pandas UDF return values or from createDataFrame(), PySpark will catch all ArrowExceptions and display info on how to disable the safe conversion config. This is displayed with the original error as a tuple:

{noformat}
('Exception thrown when converting pandas.Series (object) to Arrow Array (int32). It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled by using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`.', ArrowInvalid('Could not convert a with type str: tried to convert to int'))
{noformat}

The problem is that this is meant mainly for thing like float truncation or overflow, but will also show if the user has an invalid schema with types that are incompatible. The extra information is confusing in this case and the real error is buried.

This could be improved by only printing the extra info on how to disable safe checking if the config is actually set and using exception chaining to better show the original error. Also, any safe failures would be a ValueError, which ArrowInvaildError is a subclass, so the catch could be made more narrow.
Bug: Error executing `SHOW VIEWS in {database}` using JDBC
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Oct/20 16:02
Affects Version/s: 2.4.0

Full description: [https://stackoverflow.com/questions/60742577/spark-sql-query-show-views-in-through-hive-metastore-fails-with-missing-func]
Bug: Cannot use filter with window operations
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Oct/20 20:15
Affects Version/s: 3.0.1

Current, trying to use filter with a window operations will fail:

 
{code:java}
df = spark.range(100)
win = Window.partitionBy().orderBy('id')
df.filter(F.rank().over(win) > 10).show()
{code}
Error:
{code:java}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/icexelloss/opt/miniconda3/envs/ibis-dev-spark-3/lib/python3.8/site-packages/pyspark/sql/dataframe.py", line 1461, in filter
    jdf = self._jdf.filter(condition._jc)
  File "/Users/icexelloss/opt/miniconda3/envs/ibis-dev-spark-3/lib/python3.8/site-packages/pyspark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
  File "/Users/icexelloss/opt/miniconda3/envs/ibis-dev-spark-3/lib/python3.8/site-packages/pyspark/sql/utils.py", line 134, in deco
    raise_from(converted)
  File "<string>", line 3, in raise_from
pyspark.sql.utils.AnalysisException: It is not allowed to use window functions inside WHERE clause;{code}
Although the code is same as the code below, which works:
{code:java}
df = spark.range(100)
win = Window.partitionBy().orderBy('id')
df = df.withColumn('rank', F.rank().over(win))
df = df[df['rank'] > 10]
df = df.drop('rank'){code}
Bug: Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schema size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 01:36
Affects Version/s: 3.0.1

Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schema size

 

FOR comment [https://github.com/apache/spark/pull/29421#discussion_r511684691]
Bug: Spark 3.0 failure due to lack of Arrow dependency
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 29/Oct/20 04:06
Affects Version/s: 3.0.0

A recent change in Arrow has split the arrow-memory module into 3, so client code must add a dependency of arrow-memory-netty (or arrow-memory-unsafe).

This has been done in the master branch of Spark, but not in the branch-3.0 branch, this is causing the build in branch-3.0 to fail (https://github.com/ursa-labs/crossbow/actions?query=branch:actions-681-github-test-conda-python-3.7-spark-branch-3.0)
Bug: Intermittent Compilation failure In GitHub Actions after SBT upgrade
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/20 03:39
Affects Version/s: 3.1.0

https://github.com/apache/spark/runs/1314691686

{code}
Error:  java.util.MissingResourceException: Can't find bundle for base name org.scalactic.ScalacticBundle, locale en
Error:  	at java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:1581)
Error:  	at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1396)
Error:  	at java.util.ResourceBundle.getBundle(ResourceBundle.java:782)
Error:  	at org.scalactic.Resources$.resourceBundle$lzycompute(Resources.scala:8)
Error:  	at org.scalactic.Resources$.resourceBundle(Resources.scala:8)
Error:  	at org.scalactic.Resources$.pleaseDefineScalacticFillFilePathnameEnvVar(Resources.scala:256)
Error:  	at org.scalactic.source.PositionMacro$PositionMacroImpl.apply(PositionMacro.scala:65)
Error:  	at org.scalactic.source.PositionMacro$.genPosition(PositionMacro.scala:85)
Error:  	at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
Error:  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.lang.reflect.Method.invoke(Method.java:498)
{code}
Bug: Add asc_nulls_* and desc_nulls_* methods to SparkR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 08:51
Affects Version/s: 3.1.0

At the moment Spark provides only

- {{asc}}
- {{desc}}

but {{NULL}} handling variants

- {{asc_nulls_first}}
- {{asc_nulls_last}}
- {{desc_nulls_first}}
- {{desc_nulls_last}}

are missing.
Bug: Use new API to construct ParquetFileReader and read Parquet footer
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 27/Oct/20 03:18
Affects Version/s: 3.1.0

{code:java}
/**
 * @param configuration the Hadoop conf
 * @param fileMetaData fileMetaData for parquet file
 * @param filePath Path for the parquet file
 * @param blocks the blocks to read
 * @param columns the columns to read (their path)
 * @throws IOException if the file can not be opened
 * @deprecated will be removed in 2.0.0.
 */
@Deprecated
public ParquetFileReader(
    Configuration configuration, FileMetaData fileMetaData,
    Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns) throws IOException {
 {code}
{code:java}
/**
 * Reads the meta data block in the footer of the file
 * @param configuration a configuration
 * @param file the parquet File
 * @param filter the filter to apply to row groups
 * @return the metadata blocks in the footer
 * @throws IOException if an error occurs while reading the file
 * @deprecated will be removed in 2.0.0;
 *             use {@link ParquetFileReader#open(InputFile, ParquetReadOptions)}
 */
@Deprecated
public static final ParquetMetadata readFooter(Configuration configuration, FileStatus file, MetadataFilter filter) throws IOException
{code}
{code:java}
/**
 * Reads the meta data in the footer of the file.
 * Skipping row groups (or not) based on the provided filter
 * @param configuration a configuration
 * @param file the Parquet File
 * @param filter the filter to apply to row groups
 * @return the metadata with row groups filtered.
 * @throws IOException  if an error occurs while reading the file
 * @deprecated will be removed in 2.0.0;
 *             use {@link ParquetFileReader#open(InputFile, ParquetReadOptions)}
 */
public static ParquetMetadata readFooter(Configuration configuration, Path file, MetadataFilter filter) throws IOException{code}
 in ParquetFileReader were marked as deprecated, use 
{code:java}
public ParquetFileReader(InputFile file, ParquetReadOptions options) throws IOException
{code}
{code:java}
public ParquetMetadata getFooter()
{code}
 to instead of them.