av9ash/GitBugs huggingface dataset:
Bug: Loss of precision using SparkSQL shell on high-precision DECIMAL types
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Sep/22 16:29
Affects Version/s: 3.2.1

h3. Describe the bug

We are trying to save {{DECIMAL}} values with high precision in a table using the SparkSQL shell. When we {{INSERT}} decimal values with precision higher than the standard double precision, precision is lost. (8.8888888888888888888e9 interpreted as 8888888888.8888900000 instead of 8888888888.8888888888).

This seems to be caused by type inference at shell parsing inferring that the value is a double type.
h3. To reproduce

On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-sql{}}}:
{code:java}
$SPARK_HOME/bin/spark-sql{code}
In the shell:
{code:java}
CREATE TABLE t(c0 DECIMAL(20,10));         
INSERT INTO t VALUES (8.8888888888888888888e9);                             
SELECT * FROM t;{code}
Executing the above gives this:
{code:java}
spark-sql> CREATE TABLE t(c0 DECIMAL(20,10));
22/09/29 11:28:41 WARN ResolveSessionCatalog: A Hive serde table will be created as there is no table provider specified. You can set spark.sql.legacy.createHiveTableByDefault to false so that native data source table will be created instead.
Time taken: 0.118 seconds
spark-sql> INSERT INTO t VALUES (8.8888888888888888888e9);
Time taken: 0.392 seconds
spark-sql> SELECT * FROM t;
8888888888.8888900000
Time taken: 0.197 seconds, Fetched 1 row(s){code}
h3. Expected behavior

We expect the inserted value to retain the precision as determined by the parameters for the {{DECIMAL}} type. For example, we expect the example above to return {{{}8888888888.8888888888{}}}.
Bug: Migrate arithmetic type check failures onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 16:56
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in arithmetic expressions:
1. Least (2): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L1188-L1191
2. Greatest (2): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L1266-L1269
Bug: Add official image dockerfile for Spark v3.3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/22 12:26
Affects Version/s: 3.4.0

Example: [https://github.com/Yikun/spark-docker/tree/master/3.3.0]

Test: https://github.com/Yikun/spark-docker/blob/master/.github/workflows/build_3.3.0.yaml
Bug: Migrate collection type check failures onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 16:39
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in collection expressions:
1. BinaryArrayExpressionWithImplicitCast (1): [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L69]
2. MapContainsKey (2): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L231-L237
3. MapConcat (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L663
4. MapFromEntries (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L801
Bug: Eagerly release hashed relation in ShuffledHashJoin
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Sep/22 07:19
Affects Version/s: 3.4.0

ShuffledHashJoin releases the built hashed relation at the end of task using taskCompletionListener. It is not always good enough for complex sql query.

If a smj or window on the top of the shj, then the hashed relation in shj would be leak. All rows have been consumed in sort before smj or window then the buffer can not allocate the memory which is hold by hashed relation. Then it causes unnecessary spill.

It is a common case in multi-join, since AQE supports convert smj to shj at runtime.
Bug: Non-deterministic filters shouldn't get pushed to V2 file sources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Sep/22 12:56
Affects Version/s: 3.3.0

Currently non-deterministic filters can be pushed down to V2 file sources, which is different from V1 which prevents out non-deterministic filters from being pushed.

Main consequences:
 * Things like doing a rand filter on a partition column will throw an exception:
 ** {{IllegalArgumentException: requirement failed: Nondeterministic expression org.apache.spark.sql.catalyst.expressions.Rand should be initialized before eval.}}
 * {{Using a non-deterministic UDF to collect metrics via accumulators gets pushed down and gives the wrong metrics}}
Bug: Support double-quoted identifiers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/22 18:20
Affects Version/s: 3.4.0

In many SQL identifiers can be unquoted or quoted with double quotes. 
In Spark double quoted literals imply strings.
In this proposal we allow for a config:

double_quoted_identifiers
which, when set, switches the interpretation from string to identifier.

Note that back ticks are still allowed.
Also the treatment of escapes is not changed as part of this work.
Bug: Spark creates an optional columns in hive table for fields that are not null
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/22 18:04
Affects Version/s: 3.3.0

Dataframe saveAsTable sets all columns as optional/nullable while creating the table here  

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L531]

(`outputColumns.toStructType.asNullable`)

This makes source parquet schema and hive table schema doesn't match and is problematic when large dataframe(s) process uses hive as temporary storage to avoid the memory pressure. 

Hive 3.x supports non null constraints on table columns. Please add support for non null constraints on Spark sql hive table.
Bug: Repartition of DataFrame can result in severe data skew in some special case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/22 08:38
Affects Version/s: 3.0.1

{code:scala}
_val df = spark.range(0, 100, 1, 50).repartition(4)_
_val v = df.rdd.mapPartitions { iter => {_
        _Iterator.single(iter.length)_
{_}}{_}{_}.collect(){_}
_println(v.mkString(","))_
{code}

The above simple code outputs `50,0,0,50`, which means there is no data in partition 1 and partition 2.

I just debugged it and found the RoundRobin seems to ensure to distribute the records evenly **in the same partition**, and not guarantee it between partitions.

Below is the code to generate the key

{code:scala}
      case RoundRobinPartitioning(numPartitions) =>
        // Distributes elements evenly across output partitions, starting from a random partition.
        var position = new Random(TaskContext.get().partitionId()).nextInt(numPartitions)  
        (row: InternalRow) => {
          // The HashPartitioner will handle the `mod` by the number of partitions
          position += 1
          position
        }
{code}

In this case, There are 50 partitions, each partition will only compute 2 elements. The issue for RoundRobin here is it always starts with *position=2* to do the Roundrobin.

See the output of Random
{code:scala}
scala> (1 to 200).foreach(partitionId => print(new Random(partitionId).nextInt(4) + " "))  // the position is always 2.
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 
{code}

Similarly, the below Random code also outputs the same value, 

{code:scala}
(1 to 200).foreach(partitionId => print(new Random(partitionId).nextInt(2) + " "))

(1 to 200).foreach(partitionId => print(new Random(partitionId).nextInt(4) + " "))

(1 to 200).foreach(partitionId => print(new Random(partitionId).nextInt(8) + " "))

(1 to 200).foreach(partitionId => print(new Random(partitionId).nextInt(16) + " "))

(1 to 200).foreach(partitionId => print(new Random(partitionId).nextInt(32) + " "))
{code}

Let's go back to this case,

Consider partition 0, the total elements are [0, 1], so when shuffle writes, for element 0, the key will be (position + 1) = 2 + 1 = 3%4=3, the element 1, the key will be (position + 1)=(3+1)=4%4 = 0
consider partition 1, the total elements are [2, 3], so when shuffle writes, for element 2, the key will be (position + 1) = 2 + 1 = 3%4=3, the element 3, the key will be (position + 1)=(3+1)=4%4 = 0

 

The calculation is also applied for other left partitions since the starting position is always 2 for this case.

So, as you can see, each partition will write its elements to Partition [0, 3], which results in Partition [1, 2] without any data.

 

I will try to provide the patch to fix this issue.
Bug: Bug in MergeScalarSubqueries rule attempting to merge nested subquery with parent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/22 23:42
Affects Version/s: 3.4.0

There is a bug in the `MergeScalarSubqueries` rule for queries with subquery expressions nested inside each other, wherein the rule attempts to merge the nested subquery with its enclosing parent subquery. The result is not a valid plan and raises an exception in the optimizer. Here is a minimal reproducing case:
 
```
sql("create table test(col int) using csv")
checkAnswer(sql("select(select sum((select sum(col) from test)) from test)"), Row(null))
```
Bug: Fix test for `DataFrame.corr_with` to skip the pandas regression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/22 01:09
Affects Version/s: 3.4.0

There is a regression in pandas 1.5.0 for DataFrame.corrwith ([https://github.com/pandas-dev/pandas/issues/48826])

We should make the test pass to support pandas 1.5.x.
Bug: Support recursiveFileLookup for partitioned datasource
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Sep/22 13:12
Affects Version/s: 3.1.1

I use hive tez engine to execute union statement and insert into partitioned table may generate HIVE_UNION_SUBDIR subdirectory, and when I use spark sql to read this partitioned table, the data below HIVE_UNION_SUBDIR is not read.

For non-partitioned table, I can read the subdirectories of the table when setting recursiveFileLookup to true, but for partitioned table, it seems impossible to set recursiveFileLookup to true.

So I want to support recursiveFileLookup for partitioned table.
Bug: `V2ScanRelationPushDown` supports extracting predicates in the output set
Status: Open | Priority: Major | Resolution: 
Created: 29/Sep/22 09:52
Affects Version/s: 3.3.0

Currently does not push down `(ID = 20220921) OR (ID = 20220910)`, if `ID` is a partition field, it makes sense to implement push down
{code:java}
SELECT * FROM h2.test.people WHERE ID = 20220921 OR (ID =20220910 and split(NAME, ',')[0] == 'fred') {code}
h3. before
{code:java}
== Analyzed Logical Plan ==
NAME: string, ID: int
Project [NAME#5, ID#6]
+- Filter ((ID#6 = 20220921) OR ((ID#6 = 20220910) AND (split(NAME#5, ,, -1)[0] = fred)))
   +- SubqueryAlias h2.test.people
         +- RelationV2[NAME#5, ID#6] h2.test.people test.people

== Optimized Logical Plan ==
Filter ((ID#6 = 20220921) OR ((ID#6 = 20220910) AND (split(NAME#5, ,, -1)[0] = fred)))
+- RelationV2[NAME#5, ID#6] test.people

== Physical Plan ==
*(1) Filter ((ID#6 = 20220921) OR ((ID#6 = 20220910) AND (split(NAME#5, ,, -1)[0] = fred)))
+- *(1) Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1@56c42964 [NAME#5,ID#6] PushedFilters: [], ReadSchema: struct<NAME:string,ID:int> {code}
h3. after
{code:java}
== Analyzed Logical Plan ==
NAME: string, ID: int
Project [NAME#5, ID#6]
+- Filter ((ID#6 = 20220921) OR ((ID#6 = 20220910) AND (split(NAME#5, ,, -1)[0] = fred)))
   +- SubqueryAlias h2.test.people
         +- RelationV2[NAME#5, ID#6] h2.test.people test.people 

== Optimized Logical Plan ==
Filter ((ID#6 = 20220921) OR ((ID#6 = 20220910) AND (split(NAME#5, ,, -1)[0] = fred)))
+- RelationV2[NAME#5, ID#6] test.people

== Physical Plan ==
*(1) Filter ((ID#6 = 20220921) OR ((ID#6 = 20220910) AND (split(NAME#5, ,, -1)[0] = fred)))
+- *(1) Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1@56c42964 [NAME#5,ID#6] PushedFilters: [(ID = 20220921) OR (ID = 20220910)], ReadSchema: struct<NAME:string,ID:int>{code}
Bug: `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile`  reload after  SPARK-17321
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/22 02:38
Affects Version/s: 3.4.0

After SPARK-17321, YarnShuffleService will persist data to local shuffle state db and reload data from  local shuffle state db only when Yarn NodeManager  start with `YarnConfiguration#NM_RECOVERY_ENABLED = true` , but `YarnShuffleIntegrationSuite` not set this config and the default value of the configuration is false,  so `YarnShuffleIntegrationSuite` will neither trigger data persistence to the db nor verify the reload of data
Bug: Treat unknown partitioning as UnknownPartitioning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/22 20:08
Affects Version/s: 3.3.0

When running spark application against spark 3.3, I see the following :
{code}
java.lang.IllegalArgumentException: Unsupported data source V2 partitioning type: CustomPartitioning
    at org.apache.spark.sql.execution.datasources.v2.V2ScanPartitioning$$anonfun$apply$1.applyOrElse(V2ScanPartitioning.scala:46)
    at org.apache.spark.sql.execution.datasources.v2.V2ScanPartitioning$$anonfun$apply$1.applyOrElse(V2ScanPartitioning.scala:34)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
{code}
The CustomPartitioning works fine with Spark 3.2.1
This PR proposes to relax the code and treat all unknown partitioning the same way as that for UnknownPartitioning.
Bug: Streaming metrics is zero when select _metadata
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 00:34
Affects Version/s: 3.3.0

Streaming metrics report all 0 (`processedRowsPerSecond`, etc) when selecting `_metadata` column. Because the logical plan from the batch and the actual planned logical are mismatched: [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala#L348]
Bug: Column pruning is not handled correctly in CSV when _corrupt_record is used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 04:36
Affects Version/s: 3.2.2

I have found that depending on the name of the corrupt record in CSV, the field is populated incorrectly. Here is an example:
{code:java}
1,a > /tmp/file.csv

===

val df = spark.read
  .schema("c1 int, c2 string, x string, _corrupt_record string")
  .csv("file:/tmp/file.csv")
  .withColumn("x", lit("A"))

Result:

+---+---+---+---------------+
|c1 |c2 |x  |_corrupt_record|
+---+---+---+---------------+
|1  |a  |A  |1,a            |
+---+---+---+---------------+{code}
 

However, if you rename the {{_corrupt_record}} column to something else, the result is different:
{code:java}
val df = spark.read 
  .option("columnNameCorruptRecord", "corrupt_record")
  .schema("c1 int, c2 string, x string, corrupt_record string") 
  .csv("file:/tmp/file.csv") .withColumn("x", lit("A")) 

Result:

+---+---+---+--------------+
|c1 |c2 |x  |corrupt_record|
+---+---+---+--------------+
|1  |a  |A  |null          |
+---+---+---+--------------+{code}
 

This is due to inconsistency in CSVFileFormat, when enabling columnPruning, we check SQLConf option for corrupt records but CSV reader relies on {{columnNameCorruptRecord}} option instead.

Also, this disables column pruning which used to work in Spark version prior to https://github.com/apache/spark/commit/959694271e30879c944d7fd5de2740571012460a.
Bug: NullPointerException: Cannot invoke invalidateSerializedMapOutputStatusCache() because "shuffleStatus" is null
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/22 15:02
Affects Version/s: 3.3.0

I'm running a simple little Spark 3.3.0 pipeline on Windows 10 using Java 17 and UDFs. I hardly do anything interesting, and now when I run the pipeline on only 30,000 records I'm getting this:

{noformat}
[ERROR] Error in removing shuffle 2
java.lang.NullPointerException: Cannot invoke "org.apache.spark.ShuffleStatus.invalidateSerializedMapOutputStatusCache()" because "shuffleStatus" is null
        at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1(MapOutputTracker.scala:882)
        at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1$adapted(MapOutputTracker.scala:881)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.spark.MapOutputTrackerMaster.unregisterShuffle(MapOutputTracker.scala:881)
        at org.apache.spark.storage.BlockManagerStorageEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerStorageEndpoint.scala:59)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
        at org.apache.spark.storage.BlockManagerStorageEndpoint.$anonfun$doAsync$1(BlockManagerStorageEndpoint.scala:89)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
{noformat}

I searched and couldn't find any of the principal terms in the error message.

Disconcerting that Spark is breaking at what seems to be a fundamental part of processing, and with a {{NullPointerException}} at that.

I have already asked this question on [Stack Overflow|https://stackoverflow.com/q/73732970], and even posted a bounty, with no solutions. (The only answer so far is from someone who doesn't even use Spark and just posted links.)

_Update:_ Now it just happened with only 1000 records. But then I reran the pipeline immediately with no changes, and it succeeded. So this {{NullPointerException}} bug is nondeterministic. Not good at all.

_Update:_ Now it just happened with only 10 records. But then as before I reran the pipeline immediately with no changes, and it succeeded.
Bug: Improving testability of GroupState in applyInPandasWithState
Status: Open | Priority: Major | Resolution: 
Created: 27/Sep/22 08:21
Affects Version/s: 3.4.0

SPARK-35800 introduced a testable instance of GroupState in Scala/Java API. The rationalization was that default implementation of GroupState couldn't be instantiated.

The GroupState being introduced in PySpark is something end users can create an instance directly, but looks like the constructor is only accounting the case it comes from serialized form from executor.

It would be better to enable GroupState instance to be created by end user for testing purpose, with isolation to the constructor for internal usage.
Bug: The distributed runtime has one more identical process with a small amount of data on the master
Status: Open | Priority: Major | Resolution: 
Created: 26/Sep/22 12:26
Affects Version/s: 3.3.0

When I ran my program with the Dataframe structure in Pyspark.PANDAS, there is an abnormal extra process on the master. My dataframe contains three columns named "id", "path", and "category". It contains more than 300,000 pieces of data in total, and the "id" values are only 1, 2, 3, and 4. When I use "groupBy (" id ").apply(func)", my four nodes run normally, but there is an abnormal process in the master, which contains 1001 pieces of data. This process also executes the code in "func" and is divided into four parts, each part contains more than 200 pieces of data. When I collect the results in each node, I can only collect the results of 1001 data points, and the results of 300,000 data points are lost. When I tried to reduce the number of data to about 20,000, this problem still occurred and the data volume was still 1001. I suspect there is a problem with the implementation of this API.I tried setting the number of data partitions to 4, but the problem didn't go away.The value of the dataframe, part of the code, and the output of the exception process are attached
Bug: Migrate window type check failures onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 16:31
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in window expressions:
1. WindowSpecDefinition (4): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L68-L85
2. SpecifiedWindowFrame (3): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L216-L231
3. checkBoundary (2): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L264-L269
4. FrameLessOffsetWindowFunction (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L424
Bug: Spark Streaming support Debezium
Status: Open | Priority: Major | Resolution: 
Created: 26/Sep/22 15:00
Affects Version/s: 3.4.0

Debezuim is a very popular CDC technology. Spark Structured Streaming supports Debezuim, which facilitates data writing to data lakes。

The most commonly used scheme is FLink CDC，Hope Spark can support it
Bug: Perform maintenance of StateStore instances when they become inactive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Sep/22 04:39
Affects Version/s: 3.3.0

Current the maintenance of StateStore is performed by a periodic task in the management thread. If a streaming query become inactive before the next maintenance task fire, its StateStore will be unloaded before cleanup.
There are 2 cases when a StateStore is unloaded.
 # StateStoreProvider is not longer active in the system, for example, when a query ends or the spark context terminates.
 # There is other active StateStoreProvider in the system, for example, when a partition is reassigned.

In case 1, we should do one last maintenance before unloading the instance.
Bug: Correct CSV schema inference and data parsing behavior on columns with mixed dates and timestamps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 17:36
Affects Version/s: 3.4.0

In this ticket https://issues.apache.org/jira/browse/SPARK-39469, we introduced the support of date type in CSV schema inference. The schema inference behavior on date time columns now is:
 * For a column only containing dates, we will infer it as Date type
 * For a column only containing timestamps, we will infer it as Timestamp type
 * For a column containing a mixture of dates and timestamps, we will infer it as Timestamp type

However, we found that we are too ambitious on the last scenario, to support which we have introduced much complexity in code and caused a lot of performance concerns. Thus, we want to simplify and correct the behavior of the last scenario as:
 * For a column containing a mixture of dates and timestamps
 ** If user specifies timestamp format, it will always be inferred as `StringType`
 ** If no timestamp format specified by user, we will try inferring it as `TimestampType` if possible, otherwise it will be inferred as `StringType`