av9ash/GitBugs huggingface dataset:
Bug: Sort the whole-stage codegen debug output by codegenStageId
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Mar/20 06:28
Affects Version/s: 2.4.0

Spark SQL's whole-stage codegen (WSCG) supports dumping the generated code to help with debugging. One way to get the generated code is through {{df.queryExecution.debug.codegen}}, or SQL {{explain codegen}} statement.

The generated code is currently printed without specific ordering, which can make debugging a bit annoying. This ticket tracks a minor improvement to sort the codegen dump by the {{codegenStageId}}, ascending.

After this change, the following query:
{code}
spark.range(10).agg(sum('id)).queryExecution.debug.codegen
{code}
will always dump the generated code in a natural, stable order.

The number of codegen stages within a single SQL query tends to be very small, most likely < 50, so the overhead of adding the sorting shouldn't be significant.
Bug: Add checkValue for spark.sql.session.timeZone
Status: Open | Priority: Minor | Resolution: 
Created: 04/Mar/20 15:27
Affects Version/s: 3.1.0

The `spark.sql.session.timeZone` config can accept any string value including invalid time zone ids, then it will fail other queries that rely on the time zone. We should do the value checking in the set phase and fail fast if the zone value is invalid.
Bug: Support enabling maven profiles for importing via sbt on Intellij IDEA.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Mar/20 10:45
Affects Version/s: 3.0.0

At the moment there is no easy way to enable maven profiles if the intellij IDEA project is imported via SBT. Only other work around is to set either OS level Environment variable SBT_MAVEN_PROFILES. 
So, in this patch we add a property sbt.maven.profiles, which can be configured at the time of importing spark in IntelliJ IDEA. 
See attached image for steps to set it up.
Bug: Show message for empty Streaming Queries instead of empty timelines and histograms.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/20 02:54
Affects Version/s: 3.0.0

If a Streaming Query has no batches, StreamingQueryStatisticsPage shows empty timelines and histograms. 
It's ugly and better to show a message that clearly says "No visualization information available".
Bug: Opportunity to simplify execution plan when passing empty dataframes to subtract()
Status: Resolved | Priority: Minor | Resolution: Won't Do
Created: 12/Mar/20 19:40
Affects Version/s: 2.4.5

Execution plans are similar when passing an empty versus non-empty DataFrame to pyspark's subtract call.
{code:java}
df.subtract(regDf){code}
yields the same physical plan as:
{code:java}
df.subtract(emptyDf){code}
 Since the operation (EXCEPT DISTINCT in Spark SQL) requires a sort on both DataFrames, this can yield some significant performance speed-ups because if the incoming DF is empty no processing should happen.

 

Should be a quick fix for a seasoned commiter.
Bug: Fix Uncaught TypeError in streaming statistics page
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Mar/20 01:51
Affects Version/s: 3.0.0

There is a minor issue in https://github.com/apache/spark/pull/26201
In the streaming statistics page, there is such error 
```
streaming-page.js:211 Uncaught TypeError: Cannot read property 'top' of undefined
    at SVGCircleElement.<anonymous> (streaming-page.js:211)
    at SVGCircleElement.__onclick (d3.min.js:1)
```
in the console after clicking the timeline graph.
We should fix it.
Bug: Use the same version of `commons-io` in SBT
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Mar/20 07:13
Affects Version/s: 2.4.5

Maven always uses 2.4 version while SBT uses 2.5 at `hadoop-3.2` profile.
{code}
<commons-io.version>2.4</commons-io.version>
{code}

**branch-3.0**
{code}
$ build/sbt -Phadoop-3.2 "core/dependencyTree" | grep commons-io:commons-io | head -n1
    [info]   | | +-commons-io:commons-io:2.5
{code}

**branch-2.4**
{code}
$ build/sbt -Phadoop-3.1 "core/dependencyTree" | grep commons-io:commons-io | head -n1
[info]   | | +-commons-io:commons-io:2.5
{code}
Bug: GMM compute summary and update distributions in one pass
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Mar/20 04:08
Affects Version/s: 3.1.0

In current impl, GMM need to trigger two jobs at one iteration:

1, one to compute summary;

{color:#172b4d}2, if {{{color}{color:#c7a65d}{color:#172b4d}shouldDistributeGaussians}} {color}{color}((k - {color:#4dacf0}1.0{color}) / k) * numFeatures > {color:#4dacf0}25.0,{color}

{color:#c7a65d}{color:#172b4d}trigger another to update distributions;{color}
{color}

 

{color:#c7a65d}{color:#172b4d}shouldDistributeGaussians is almost true in practice, since numFeatures is likely to be greater than 25.{color}{color}

 

{color:#c7a65d}{color:#172b4d}We can use only one job to impl above computation,{color}{color}
Bug: Add documentation for ALTER TABLE ... ADD PARTITION
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 06/Mar/20 18:29
Affects Version/s: 3.1.0

Our docs for {{ALTER TABLE}} [currently|https://github.com/apache/spark/blob/cba17e07e9f15673f274de1728f6137d600026e1/docs/sql-ref-syntax-ddl-alter-table.md] make no mention of {{ADD PARTITION}}.
Bug: Bugs/missing functions in documents
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 07/Mar/20 08:09
Affects Version/s: 2.4.5

In current document for SQL API, I noticed that there is no section for `PIVOT` keyword, which was introduced from 2.4.0.

Is there a bug in `mkdocs`? 

Docs: [https://spark.apache.org/docs/latest/api/sql/]

P/S: Not sure if this issue should be in here but I cannot found any other place to put this it.
Bug: InMemoryStore: CountingRemoveIfForEach misses to remove key from parentToChildrenMap
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/20 03:10
Affects Version/s: 3.0.0

SPARK-30964 introduces the secondary index which defines the relationship between parent - children and able to operate all children for given parent faster.

This change is not applied to CountingRemoveIfForEach, so there's a chance "countingRemoveAllByIndexValues" missed to remove key from parentToChildrenMap.
Bug: Support foldable schemas by `from_csv`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/20 14:32
Affects Version/s: 3.0.0

Currently, Spark accepts only literals or schema_of_csv w/ literal input as the schema parameter of from_csv. And it fails on any foldable expressions, for instance:
{code:sql}
spark-sql> select from_csv('1, 3.14', replace('dpt_org_id INT, dpt_org_city STRING', 'dpt_org_', ''));
Error in query: Schema should be specified in DDL format as a string literal or output of the schema_of_csv function instead of replace('dpt_org_id INT, dpt_org_city STRING', 'dpt_org_', '');; line 1 pos 7
{code}
There are no reasons to restrict users by literals. The ticket aims to support any foldable schemas by from_csv().
Bug: InMemoryStore: improve removeAllByIndexValues over natural key index
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/20 20:26
Affects Version/s: 3.1.0

The method `removeAllByIndexValues` in KVStore is to delete all the objects which have certain values in the given index.
However, in the current implementation of  `InMemoryStore`, when the given index is the natural key index, there is no special handling for it and a linear search over all the task data is performed.

We can improve it by deleting the natural keys directly from the internal hashmap.
Bug: Spark Kubernetes add tolerations and nodeName support
Status: Open | Priority: Trivial | Resolution: 
Created: 17/Mar/20 14:28
Affects Version/s: 2.4.6

When you run spark on serverless kubernetes cluster(virtual-kubelet). you need to specific the nodeSelectors,tolerations even nodeName when you want to gain better scheduling performance. Currently spark doesn't support tolerations. If you want to use this feature, You must use admission controller webhook to decorate the pod. But the performance is extremely bad. Here is the benchmark. 

With webhook 

Batch Size: 500 Pod creation: about 7 Pods/s   All Pods running: 5min

Without webhook 

Batch Size: 500 Pod creation: more than 500 Pods/s All Pods running: 45s

Adding tolerations and nodeName in spark will bring great help when you want to run a large scale job on serverless kubernetes cluster.
Bug: Update py code to generate data in testsuites
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 23/Mar/20 05:50
Affects Version/s: 3.1.0

in FValueTestSuite/ANOVASelectorSuite/ANOVATestSuite/...:

 

can not regenerate the test datasets with given py code (like {color:#676773}X = np.random.rand(20, 6){color}), so:

1, directly create X like : X = np.array(...);

2, or, set a seed at first;
Bug: Show Maven errors from within make-distribution.sh
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Mar/20 17:36
Affects Version/s: 3.1.0

This works:
{code:java}
./dev/make-distribution.sh \
 --pip \
 -Phadoop-2.7 -Phive -Phadoop-cloud {code}
 
 But this doesn't:
{code:java}
 ./dev/make-distribution.sh \
 -Phadoop-2.7 -Phive -Phadoop-cloud \
 --pip{code}
 

The latter invocation yields the following, confusing output:
{code:java}
 + VERSION=' -X,--debug Produce execution debug output'{code}
 That's because Maven is accepting {{--pip}} as an option and failing, but the user doesn't get to see the error from Maven.
Bug: Job cancellation causes py4j errors on Jupyter due to pinned thread mode
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 14/Oct/21 05:15
Affects Version/s: 3.2.0

Spark 3.2.0 turned on py4j pinned thread mode by default (SPARK-35303). However, in a jupyter notebook, after I cancel (interrupt) a long-running Spark job, the next Spark command will fail with some py4j errors. See attached notebook for repro.

Cannot reproduce the issue after I turn off pinned thread mode .
Bug: Disable pinned thread mode by default
Status: Resolved | Priority: Blocker | Resolution: Invalid
Created: 15/Oct/21 01:52
Affects Version/s: 3.3.0

Pinned thread mode was enabled by default in Spark 3.2 (SPARK-35303). However, it causes some breaking changes such as SPARK-37004. Maybe we should disable it by default for now.
Bug: Report driver status does not handle response from backup masters
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 19/Oct/21 14:29
Affects Version/s: 3.1.0

After an improvement in SPARK-31486, contributor uses 'asyncSendToMasterAndForwardReply' method instead of 'activeMasterEndpoint.askSync' to get the status of driver. Since the driver's status is only available in active master and the 'asyncSendToMasterAndForwardReply' method iterate over all of the masters, we have to handle the response from the backup masters in the client, which the developer did not consider in the SPARK-31486 change. So drivers running in cluster mode and on a cluster with multi-master affected by this bug. I created the patch for this bug and will soon be sent the pull request.
Bug: Add Codegen support to array higher-order-functions
Status: In Progress | Priority: Major | Resolution: 
Created: 15/Oct/21 11:40
Affects Version/s: 3.2.0

Currently all of the higher order functions use CodegenFallback. We can improve the performance of these by adding proper codegen support, so the function as well as all children can be codegen'd, and it can participate in WholeStageCodegen.

This ticket is for adding support to array-based higher-order functions.
Bug: Upgrade libthrift to resolve security vulnerabilities
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 21/Oct/21 19:58
Affects Version/s: 3.0.0

Currently, Spark uses libthrift 0.12, which has reported high severity security vulnerabilities https://snyk.io/vuln/maven:org.apache.thrift%3Alibthrift
Upgrade to 0.14 to get rid of vulnerabilities.
Bug: Migrate type hint data tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/21 19:18
Affects Version/s: 3.3.0

Before the migration, {{pyspark-stubs}} contained a set of [data tests|https://github.com/zero323/pyspark-stubs/tree/branch-3.0/test-data/unit], modeled after, and using internal test utilities, of mypy.

These were omitted during the migration for a few reasons:
 * Simplicity.
 * Relative slowness.
 * Dependence on non public API.

 

Data tests are useful for a number of reasons:

 
 * Improve test coverage for type hints.
 * Checking if type checkers infer expected types.
 * Checking if type checkers reject incorrect code.
 * Detecting unusual errors with code that otherwise type checks,

 

Especially, the last two functions are not fulfilled by simple validation of existing codebase.

 

Data tests are not required for all annotations and can be restricted to code that has high possibility of failure:
 * Complex overloaded signatures.
 * Complex generics.
 * Generic {{self}} annotations
 * Code containing {{type: ignore}}

The biggest risk, is that output matchers have to be updated when signature changes and / or mypy output changes.

Example of problem detected with data tests can be found in SPARK-36894 PR ([https://github.com/apache/spark/pull/34146]).
Bug: Upgrade Apache Thrift
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 13/Oct/21 06:36
Affects Version/s: 3.0.1

*Image:*
 * spark:3.0.1

*Components Affected:* 
 * Apache Thrift

*Recommendation:*
 * upgrade Apache Thrift 

*CVE:*

 
|Component Name|Component Version Name|Vulnerability|Fixed version|
|Apache Thrift|0.11.0-4.|CVE-2019-0205|0.13.0|
|Apache Thrift|0.11.0-4.|CVE-2019-0210|0.13.0|
|Apache Thrift|0.11.0-4.|CVE-2020-13949|0.14.1|
Bug: np.nan series.astype(bool) should be True
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/21 06:48
Affects Version/s: 3.3.0

np.nan series.astype(bool) should be True, rather than Fasle:

https://github.com/apache/spark/blob/46bcef7472edd40c23afd9ac74cffe13c6a608ad/python/pyspark/pandas/data_type_ops/base.py#L147

>>> pd.Series([1, 2, np.nan], dtype=float).astype(bool)
>>> pd.Series([1, 2, np.nan], dtype=str).astype(bool)
>>> pd.Series([datetime.date(1994, 1, 31), datetime.date(1994, 2, 1), np.nan])
0     True
1     True
2     True
dtype: bool

But in pyspark, it is:
0     True
1     True
2     False
dtype: bool
Bug: Introduce a rank-based filter to optimize top-k computation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/21 10:47
Affects Version/s: 3.4.0

in JD, we found that more than 90% usage of window function follows this pattern:
{code:java}
 select (... (row_number|rank|dense_rank) () over( [partition by ...] order by ... ) as rn)
    where rn (==|<|<=) k and other conditions{code}
 

However, existing physical plan is not optimum:

 

1, we should select local top-k records within each partitions, and then compute the global top-k. this can help reduce the shuffle amount;

 

For these three rank functions (row_number|rank|dense_rank), the rank of a key computed on partitial dataset  is always <=  its final rank computed on the whole dataset. so we can safely discard rows with partitial rank > k, anywhere.

 

 

2, skewed-window: some partition is skewed and take a long time to finish computation.

 

A real-world skewed-window case in our system is attached.
Bug: Apply 'compute.eager_check' across all the codebase
Status: Open | Priority: Major | Resolution: 
Created: 19/Oct/21 06:00
Affects Version/s: 3.3.0

As [~hyukjin.kwon] guide


 1 Make every input validation like this covered by the new configuration. For example:
{code:python}
- a == b
+ def eager_check(f): # Utility function 
+ return not config.compute.eager_check and f() 
+ 
+ eager_check(lambda: a == b)
{code}
2 We should check if the output makes sense although the behaviour is not matched with pandas'. If the output does not make sense, we shouldn't cover it with this configuration.

3 Make this configuration enabled by default so we match the behaviour to pandas' by default.

 

We have to make sure listing which API is affected in the description of 'compute.eager_check'
Bug: Ensure the element type of ResolvedRFormula.terms is scala.Seq for Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/21 02:59
Affects Version/s: 3.3.0

ResolvedRFormula.toString throws ClassCastException with Scala 2.13 because the type of ResolvedRFormula.terms is scala.Seq[scala.Seq[String]] but scala.Seq[scala.collection.mutable.ArraySeq$ofRef] will be passed.
Bug: Query files directly with SQL is broken (with Glue)
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 11/Oct/21 06:39
Affects Version/s: 3.1.2

This is broken in DBR 9.1 (and 10.0 Beta):

{{    select * from json.`filename`}}

[https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-file.html]

I have tried with JSON and Parquet files.

The error:

{color:#FF0000}{{Error in SQL statement: SparkException: Unable to fetch tables of db json}}{color}

Down in the stack trace this also exists:

{{{color:#FF0000}Caused by: NoSuchObjectException(message:Database json not found. (Service: AWSGlue; Status Code: 400; Error Code: EntityNotFoundException; ... )){color}}}
Bug: Timestamp are written as array bytes.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/Oct/21 20:03
Affects Version/s: 3.3.0

This is tested with master build 04.10.21

{code}
df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
                   'month': [2, 3],
                   'day': [4, 5],
                  'test': [1, 2]})  

df["year"] = ps.to_datetime(df["year"]) 

df.info() 

<class 'pyspark.pandas.frame.DataFrame'> Int64Index: 2 entries, 0 to 1 Data columns (total 4 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 year 2 non-null datetime64 1 month 2 non-null int64 2 day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)  


spark_df_date = df.to_spark() 

spark_df_date.printSchema() 

root
|-- year: timestamp (nullable = true)
|-- month: long (nullable = false)
|-- day: long (nullable = false)
|-- test: long (nullable = false)  

spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")  
{code}

Load the files in to Apache drill I use docker apache/drill:master-openjdk-14  

SELECT * FROM cp.`/data/spark_df_date.*`  

It print's

year

{code}
\x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00

\x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00 
{code}
 

The rest of the columns are ok.   
So is this a spark problem or Apache drill?
Bug: Use black as a formatter for the whole PySpark codebase.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/21 20:22
Affects Version/s: 3.3.0

[{{black}}|https://github.com/psf/black] is a popular Python code formatter. It is used by a number of projects, both small and large, including prominent ones, like pandas, scikit-learn, Django or SQLAlchemy. Black is already used to format a {{pyspark.pandas}} and (though not enforced) stubs files.

We should consider using black to enforce formatting of all PySpark files. There are multiple reasons to do that:
 - Consistency: black is already used across existing codebase and black formatted chunks of code are already added to modules other than pyspark.pandas as a result of type hints inlining (SPARK-36845).
 - Lower cost of contributing and reviewing: Formatting can be automatically enforced and applied.
 - Simplify reviews: In general, black formatted code, produces small and highly readable diffs.
 - Reduce effort required to maintain patched forks: smaller diffs + predictable formatting.

Risks:
 - Initial reformatting requires quite significant changes.
 - Applying black will break blame in GitHub UI (for git in general see [Avoiding ruining git blame|https://black.readthedocs.io/en/stable/guides/introducing_black_to_your_project.html?highlight=blame#avoiding-ruining-git-blame]).

Additional steps:
 - To simplify backporting, black will have to be applied to all active branches.
Bug: spark-hadoop-cloud broken on release and only published via 3rd party repositories
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/21 05:09
Affects Version/s: 3.1.1

The spark docmentation suggests using `spark-hadoop-cloud` to read / write from S3 in [https://spark.apache.org/docs/latest/cloud-integration.html] . However artifacts are currently published via only 3rd party resolvers in [https://mvnrepository.com/artifact/org.apache.spark/spark-hadoop-cloud] including Cloudera and Palantir.

 

Then apache spark documentation is providing a 3rd party solution for object stores including S3. Furthermore, if you follow the instructions and include one of the 3rd party jars IE the Cloudera jar with the spark 3.1.2 release and try to access object store, the following exception is returned.

 

```

Exception in thread "main" java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkArgument(boolean, java.lang.String, java.lang.Object, java.lang.Object)'
 at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:894)
 at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:870)
 at org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1605)
 at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:363)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
 at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
 at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
 at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
 at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
 at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
 at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:46)
 at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:377)
 at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
 at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
 at scala.Option.getOrElse(Option.scala:189)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
 at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:519)
 at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:428)

```

It looks like there are classpath conflicts using the cloudera published `spark-hadoop-cloud` with spark 3.1.2, again contradicting the documentation.

Then the documented `spark-hadoop-cloud` approach to using object stores is poorly supported only by 3rd party repositories and not by the released apache spark whose documentation refers to it.

Perhaps one day apache spark will provide tested software so that developers can quickly and easily access cloud object stores using the documentation.
Bug: Support ps.MultiIndex.dtypes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/21 04:31
Affects Version/s: 3.3.0

when MultiIndex.dtypes is supported, we can use:
{code:java}
>>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 5, 6, 7, 8, 9]], names=("zero", "one"))
>>> pdf = pd.DataFrame(
...     {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
...     index=idx,
... )
>>> psdf = ps.from_pandas(pdf)

>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, pyspark.pandas.typedef.typehints.IndexNameType, pyspark.pandas.typedef.typehints.NameType, pyspark.pandas.typedef.typehints.NameType]
{code}
Bug: Support ANSI intervals by ABS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Oct/21 09:37
Affects Version/s: 3.3.0

Extend the Abs expression to support ANSI intervals: YearMonthIntervalType and DayTimeIntervalType. For instance:

{code:sql}
spark-sql> select abs(interval -'1-1' year to month);
1-1
{code}