av9ash/GitBugs huggingface dataset:
Bug: Lead/Lag window function throws AnalysisException without ORDER BY clause
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Feb/20 07:56
Affects Version/s: 3.1.0

 Lead/Lag window function throws AnalysisException without ORDER BY clause:
{code:java}
SELECT lead(ten, four + 1) OVER (PARTITION BY four), ten, four
FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s
org.apache.spark.sql.AnalysisException
Window function lead(ten#x, (four#x + 1), null) requires window to be ordered, please add ORDER BY clause. For example SELECT lead(ten#x, (four#x + 1), null)(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table;
{code}
 

Maybe we need fix this issue.
Bug: GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has fixed length
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Feb/20 14:11
Affects Version/s: 2.3.0

This is reported by user mailing list, though the mail thread is regarding suspect of the behavior of mapGroupsWithState.

[https://lists.apache.org/thread.html/r08b44a7afac4e4c971633d30b4e5d11bd7c0d6e28180e03b874ea58b%40%3Cuser.spark.apache.org%3E]

The actual culprit is, there're a couple of methods which don't handle UDT and it makes GenerateUnsafeRowJoiner to generate incorrect code. Specifically, the issue occurs when the sql type of UDT has fixed length - GenerateUnsafeRowJoiner has the logic to update the offset position for all variable-length data, and due to this bug, UDT field with fixed length is being treated as variable-length data and its value is modified.
Bug: months_between wrong calculation
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 05/Feb/20 18:40
Affects Version/s: 2.4.4

months_between not calculating right for February

example

 

{{select }}

{{ months_between('2020-02-29','2019-12-29')}}

{{,months_between('2020-02-29','2019-12-30') }}

{{,months_between('2020-02-29','2019-12-31') }}
 

will generate a result like this 
|2|1.96774194|2|
 
For 2019-12-30 is calculating wrong.
Bug: TABLE.COLUMN reference doesn't work with new columns created by UDF
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 28/Feb/20 18:20
Affects Version/s: 2.4.4

When a dataframe is created with an alias (`.as("...")`) its columns can be referred as `TABLE.COLUMN` but it doesn't work for newly created columns with UDF.
{code:java}

// code placeholder
df1 = sc.parallelize(l).toDF("x","y").as("cat")
val squared = udf((s: Int) => s * s)
val df2 = df1.withColumn("z", squared(col("y")))
df2.columns //Array[String] = Array(x, y, z)

df2.select("cat.x") // works

df2.select("cat.z") // Doesn't work
// org.apache.spark.sql.AnalysisException: cannot resolve '`cat.z`' given input 
// columns: [cat.x, cat.y, z];;
{code}
Might be related to: https://issues.apache.org/jira/browse/SPARK-30532
Bug: Executing the example on https://spark.apache.org/docs/latest/running-on-yarn.html fails
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 04/Feb/20 07:48
Affects Version/s: 2.4.4

running

 {{./bin/spark-submit --class org.apache.spark.examples.SparkPi
    --master yarn
    --deploy-mode cluster
    --driver-memory 4g
    --executor-memory 2g
    --executor-cores 1
    --queue thequeue
    examples/jars/spark-examples*.jar
    10}}

results in

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
 at java.lang.Class.getDeclaredMethods0(Native Method)
 at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
 at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
 at java.lang.Class.getMethod0(Class.java:3018)
 at java.lang.Class.getMethod(Class.java:1784)
 at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
 at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
 at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
 ... 7 more{{}}

Just install slf4j or is there a way around?{{}}
Bug: Spark SQL can't display bracketed comments well in generated golden files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Feb/20 12:49
Affects Version/s: 3.0.0

Although Spark SQL support bracketed comments, but {{SQLQueryTestSuite}} can't treat bracketed comments well and lead to generated golden files can't display bracketed comments well.

We can read the output of comments.sql

[https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-tests/results/postgreSQL/comments.sql.out]

Such as:

 
{code:java}
-- !query/* This is an example of SQL which should not execute: * select 'multi-line'-- !query schemastruct<>-- !query outputorg.apache.spark.sql.catalyst.parser.ParseException
mismatched input '/' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)
== SQL ==/* This is an example of SQL which should not execute:^^^ * select 'multi-line'

-- !query*/SELECT 'after multi-line' AS fifth-- !query schemastruct<>-- !query outputorg.apache.spark.sql.catalyst.parser.ParseException
extraneous input '*/' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)
== SQL ==*/^^^SELECT 'after multi-line' AS fifth
{code}
Bug: ANSI SQL: LAST_VALUE function
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 04/Feb/20 10:12
Affects Version/s: 3.0.0

I have checked PostgreSQL, Vertica, Oracle, Redshift, Presto, Teradata, FIRST_VALUE|LAST_VALUE is always used as a window function, not as an aggregate function.

The FIRST_VALUE function currently provided can be used as both an aggregation function and a window function.

Maybe we need to re-implement it.

Reference discussion in https://github.com/apache/spark/pull/25082
Bug: ANSI SQL: FIRST_VALUE function
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 04/Feb/20 10:11
Affects Version/s: 3.0.0

I have checked PostgreSQL, Vertica, Oracle, Redshift, Presto, Teradata, FIRST_VALUE|LAST_VALUE is always used as a window function, not as an aggregate function.

The FIRST_VALUE function currently provided can be used as both an aggregation function and a window function.

Maybe we need to re-implement it.

Reference discussion in https://github.com/apache/spark/pull/25082
Bug: Application failed due to failed to get MapStatuses broadcast
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Feb/20 09:36
Affects Version/s: 2.1.0

Currently, we encountered an issue in Spark2.1. The exception is as follows:


{noformat}
	Job aborted due to stage failure: Task 18 in stage 2.0 failed 4 times, most recent failure: Lost task 18.3 in stage 2.0 (TID 13819, xxxx , executor 8): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_9_piece1 of broadcast_9
java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_9_piece1 of broadcast_9
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1287)
	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:206)
	at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:66)
	at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:66)
	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:96)
	at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
	at org.apache.spark.MapOutputTracker$$anonfun$deserializeMapStatuses$1.apply(MapOutputTracker.scala:775)
	at org.apache.spark.MapOutputTracker$$anonfun$deserializeMapStatuses$1.apply(MapOutputTracker.scala:775)
	at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
	at org.apache.spark.MapOutputTracker$.logInfo(MapOutputTracker.scala:712)
	at org.apache.spark.MapOutputTracker$.deserializeMapStatuses(MapOutputTracker.scala:774)
	at org.apache.spark.MapOutputTrackerWorker.getStatuses(MapOutputTracker.scala:665)
	at org.apache.spark.MapOutputTrackerWorker.getMapSizesByExecutorId(MapOutputTracker.scala:603)
	at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:57)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109)
{noformat}

I looked into the code and the logs, it seems that it's caused by the mapStatuses broadcast id is sent to executor, but was invalidated immediately by the driver before the real fetching of the broadcast.

This can be described as follows:

{noformat}
Let's say we have an rdd1,
rdd2 = rdd1.repartition(100) // stage 0
rdd3 = rdd2.map(xxx)  // stage 1
rdd4 = rdd2.map(xxx)  // stage 2
// and then do some join and output result
rdd3.join(rdd4).save
{noformat}


When FetchFailedException happened in stage 1, then stage 0 and stage 1 will be resubmitted and re-executed, but stage 2 is still running, it's task will fetch mapStatuses from driver, but the mapStatuses cache will be invalidated when tasks of stage 0.1 completes and registerMapOutput.

I checked the master branch, seems that we are fixed correctness issues on `repartition`, but I think this issue may still exist? 

Some ScreenShot:
!https://issues.apache.org/jira/secure/attachment/12993652/image-2020-02-16-11-17-32-103.png!
!https://issues.apache.org/jira/secure/attachment/12993651/image-2020-02-16-11-13-18-195.png!
Bug: Propagate SPARK_CONF_DIR files to driver and exec pods.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/20 10:48
Affects Version/s: 3.1.0

SPARK_CONF_DIR hosts configuration files like, 
 1) spark-defaults.conf - containing all the spark properties.
 2) log4j.properties - Logger configuration.
 3) spark-env.sh - Environment variables to be setup at driver and executor.
 4) core-site.xml - Hadoop related configuration.
 5) fairscheduler.xml - Spark's fair scheduling policy at the job level.
 6) metrics.properties - Spark metrics.
 7) Any user specific - library or framework specific configuration file.

Traditionally, SPARK_CONF_DIR has been the home to all user specific configuration files.

So this feature, will let the user specific configuration files be mounted on the driver and executor pods' SPARK_CONF_DIR.

Please review the attached design doc, for more details.

 

[Google docs link|https://bit.ly/spark-30985]
Bug: Executor pods with multiple containers will not be rescheduled unless all containers fail
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/20 20:32
Affects Version/s: 3.1.0

Since the restart policy of launched pods is Never, additional handling is required for pods that may have sidecar containers. The executor should be considered failed if any containers have terminated and have a non-zero exit code, but Spark currently only checks the pod phase. The pod phase will remain "running" as long as _any_ pods are still running. Kubernetes sidecar support in 1.18/1.19 does not address this situation, as sidecar containers are excluded from pod phase calculation.
Bug: Execute the TIMESTAMP roadmap
Status: Open | Priority: Major | Resolution: 
Created: 20/Feb/20 19:27
Affects Version/s: 2.4.5

This issue is intended for tracking the addition and/or alteration of different TIMESTAMP types in order to eventually reach the desired state as specified in the [design doc|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit] for TIMESTAMP types.

It's a sister issue to HIVE-21348 & IMPALA-9408 - I found no comparable issue for Spark (and I was hoping to find out the status of this roadmap on the Spark-side) - and related to SPARK-26797.
Bug: FileStreamSourceLog/FileStreamSinkLog: leverage UnsafeRow type to serialize/deserialize entry
Status: Closed | Priority: Major | Resolution: Won't Fix
Created: 25/Feb/20 13:51
Affects Version/s: 3.1.0

HDFSMetadataLog and its descendants are normally using JSON serialization to serialize/deserialize entries.

While it's good to support backward compatibility (like field addition and field deletion), it tends to take bunch of overhead as it adds field names, and should store all data types to string (at least when it's being written to file), works badly for some kind of fields - e.g. timestamp.

The major overhead is heavily affecting to CompactibleFileStreamLog, as "compact" operation requires to load all entities and do the transformation/filtering (I haven't seen any effective operation being implemented though), and store them altogether into one file. This is the one of major reason why the metadata file is too huge and it brings unacceptable latency on "compact" operation.

Fortunately, the entity class for both FileStreamSourceLog (FileEntry) and FileStreamSinkLog (SinkFileStatus) haven't been modified for over 3 years. The latest modification of both classes were year 2016. We can call it "stable" - and then we have more option to optimize serde.

One easy but pretty effective approach on optimizing serde is converting to UnsafeRow and storing it on the same way we do in HDFSBackedStateStoreProvider, and vice versa. It has being running for 2.x versions, so the approach is considered as safe.
Bug: Support 'like any' and 'like all' operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/20 08:33
Affects Version/s: 3.1.0

In Teradata/Hive and PostgreSQL 'like any' and 'like all' operators are mostly used when we are matching a text field with numbers of patterns. For example:

Teradata / Hive 3.0:
{code:sql}
--like any
select 'foo' LIKE ANY ('%foo%','%bar%');

--like all
select 'foo' LIKE ALL ('%foo%','%bar%');
{code}

PostgreSQL:

{code:sql}
-- like any
select 'foo' LIKE ANY (array['%foo%','%bar%']);

-- like all
select 'foo' LIKE ALL (array['%foo%','%bar%']);
{code}
Bug: AnalysisException that window RangeFrame not match RowFrame
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 05/Feb/20 02:38
Affects Version/s: 3.1.0

 

 
{code:java}
select last(salary) over(order by salary range between 1000 preceding and 1000 following),
lag(salary) over(order by salary range between 1000 preceding and 1000 following),
salary from empsalary
org.apache.spark.sql.AnalysisException
Window Frame specifiedwindowframe(RangeFrame, -1000, 1000) must match the required frame specifiedwindowframe(RowFrame, -1, -1);
{code}
 

Maybe we need to fix this issue.
Bug: Add parameter position for REGEXP_REPLACE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/20 05:32
Affects Version/s: 3.1.0

 
vertica

REGEXP_REPLACE( _string_, _target_ [, _replacement_ [, _position_ [, _occurrence_ ... [, _regexp_modifiers_ ] ] ] ] )

reference: [https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/RegularExpressions/REGEXP_REPLACE.htm?zoom_highlight=regexp_replace]

oracle

[https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/REGEXP_REPLACE.html#GUID-EA80A33C-441A-4692-A959-273B5A224490]

redshift

https://docs.aws.amazon.com/redshift/latest/dg/REGEXP_REPLACE.html
Bug: Fix flaky "Test basic decommissioning" test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/20 07:12
Affects Version/s: 3.1.0

- https://github.com/apache/spark/pull/27721
{code}
- Test basic decommissioning *** FAILED ***
  The code passed to eventually never returned normally. Attempted 126 times over 2.0100952450666667 minutes. Last failure message: "++ id -u
{code}
Bug: Driver cores in kubernetes are coupled with container resources, not spark.driver.cores
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/20 16:36
Affects Version/s: 3.1.0

Drivers submitted in kubernetes cluster mode set the parallelism of various components like 'RpcEnv', 'MemoryManager', 'BlockManager' from inferring the number of available cores by calling:
{code:java}
Runtime.getRuntime().availableProcessors()
{code}
By using this, spark applications running on java 8 or older incorrectly get the total number of cores in the host, ignoring the cgroup limits set by kubernetes (https://bugs.openjdk.java.net/browse/JDK-6515172). Java 9 and newer runtimes do not have this problem.

Orthogonal to this, it is currently not possible to decouple resource limits on the driver container with the amount of parallelism of the various network and memory components listed above.

My proposal is to use the 'spark.driver.cores' configuration to get the amount of parallelism, like we do for YARN (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L2762-L2767). This will enable users to specify 'spark.driver.cores' to set parallelism, and specify 'spark.kubernetes.driver.requests.cores' to limit the resource requests of the driver container. Further, this will remove the need to call 'availableProcessors()', thus the same number of cores will be used for parallelism independent of the java runtime version.
Bug: FileStreamSource: Cache fetched list of files beyond maxFilesPerTrigger as unread files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/20 12:50
Affects Version/s: 3.1.0

FileStreamSource fetches the available files per batch which is a "heavy cost" operation.

(E.g. It took around 5 seconds to list leaf files for 95 paths which contain 674,811 files. It's not even in HDFS path - it's local filesystem.)

If "maxFilesPerTrigger" is not set, Spark would consume all the fetched files. After the batch has been completed, it's obvious for Spark to fetch per micro batch.

If "latestFirst" is true (regardless of "maxFilesPerTrigger"), the files to process should be updated per batch, so it's also obvious for Spark to fetch per micro batch.

Except above cases (in short, maxFilesPerTrigger is being set and latestFirst is false), the files to process can be "continuous" - we can "cache" the fetched list of files and consume until the list has been exhausted.
Bug: Able to write parquet file subsequently unable to read
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 29/Feb/20 21:25
Affects Version/s: 2.4.4

{code:java}
case class StructKey(i: Int)
case class StructValue(l: Long)
case class Outer(m: Map[StructKey, StructValue])

val data = Seq(Seq(Outer(Map(StructKey(0) -> StructValue(1L)))))
val ds = data.toDS

ds.write.mode("overwrite").parquet("ds.parquet")
val in = spark.read.parquet("ds.parquet")

ds.printSchema

root
 |-- value: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- m: map (nullable = true)
 |    |    |    |-- key: struct
 |    |    |    |    |-- i: integer (nullable = false)
 |    |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |    |-- l: long (nullable = false)


ds.show(false)

+----------------+
|value           |
+----------------+
|[[[[0] -> [1]]]]|
+----------------+

in.printSchema

root
 |-- value: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- m: map (nullable = true)
 |    |    |    |-- key: struct
 |    |    |    |    |-- i: integer (nullable = true)
 |    |    |    |-- value: struct (valueContainsNull = true)
 |    |    |    |    |-- l: long (nullable = true)

in.show(false)

Caused by: org.apache.spark.sql.AnalysisException: Map key type is expected to be a primitive type, but found: required group key {
  required int32 i;
};
  at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$.checkConversionRequirement(ParquetSchemaConverter.scala:583)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter$$anonfun$convertGroupField$2.apply(ParquetSchemaConverter.scala:228)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter$$anonfun$convertGroupField$2.apply(ParquetSchemaConverter.scala:183)
  at scala.Option.fold(Option.scala:158)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter.convertGroupField(ParquetSchemaConverter.scala:183)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter.convertField(ParquetSchemaConverter.scala:90)
{code}
You should not be able to write something you subsequently can't read; if attempting to write invalid parquet it should error at write, otherwise it should successfully read.
Bug: Refactor DateTimeUtils
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/20 06:28
Affects Version/s: 3.1.0

* Move TimeZoneUTC and TimeZoneGMT to DateTimeTestUtils
* Remove TimeZoneGMT because it is equal to UTC
* Use ZoneId.systemDefault() instead of defaultTimeZone().toZoneId
* Alias SQLDate & SQLTimestamp to internal types of DateType and TimestampType
Bug: spark-submit pyspark app on yarn uploads local pyspark archives
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/20 20:59
Affects Version/s: 2.4.0

Use spark-submit to submit a pyspark app on Yarn, and set this in spark-env.sh:

{code:bash}
export PYSPARK_ARCHIVES_PATH=local:/opt/spark/python/lib/pyspark.zip,local:/opt/spark/python/lib/py4j-0.10.7-src.zip
{code}

You can see that these local archives are still uploaded to Yarn distributed cache.

yarn.Client: Uploading resource file:/opt/spark/python/lib/pyspark.zip -> hdfs://myhdfs/user/test1/.sparkStaging/application_1581024490249_0001/pyspark.zip
Bug: Upgrade fabric8 to 4.7.1 to support K8s 1.17
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/20 15:20
Affects Version/s: 3.0.0

Fabric8 kubernetes-client introduced a regression in building Quantity values in 4.7.0.

More info: [https://github.com/fabric8io/kubernetes-client/issues/1953]

As part of this upgrade, creation of quantity objects should be changed in order to keep correctly parsing quantities with units.
Bug: FileStreamSinkLog: Avoid reading the metadata log file when finding the latest batch ID
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/20 08:08
Affects Version/s: 3.1.0

FileStreamSink.addBatch checks the latest batch ID before writing outputs to skip writing batch if the batch was committed before.

While it's valid to compare the current batch with the latest batch ID, getLatest() method is designed to return both the batch ID as well as content which denotes that the latest metadata log file is being read and deserialized. This would introduces heavy latency when the latest batch is a compacted batch.

We could just find the metadata log file for latest batch ID, and only do the minimal check without reading content.