av9ash/GitBugs huggingface dataset:
Bug: Reading wrong timestamps from dictionary encoded columns in Parquet files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/20 11:38
Affects Version/s: 3.0.0

Write dates with dictionary encoding enabled to parquet files:
{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.conf.set("spark.sql.legacy.parquet.rebaseDateTimeInWrite.enabled", true)
scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")

scala> :paste
// Entering paste mode (ctrl-D to finish)

                Seq.tabulate(8)(_ => "1001-01-01 01:02:03.123").toDF("tsS")
                  .select($"tsS".cast("timestamp").as("ts"))
                  .repartition(1)
                  .write
                  .option("parquet.enable.dictionary", true)
                  .mode("overwrite")
                  .parquet("/Users/maximgekk/tmp/parquet-ts-dict")

// Exiting paste mode, now interpreting.
{code}

Read them back:
{code:scala}
scala> spark.read.parquet("/Users/maximgekk/tmp/parquet-ts-dict").show(false)
+-----------------------+
|ts                     |
+-----------------------+
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
|1001-01-07 00:32:20.123|
+-----------------------+
{code}

*Expected values must be 1001-01-01 01:02:03.123.*

I checked that the timestamp column is encoded by dictionary via:
{code}
➜  parquet-ts-dict java -jar ~/Downloads/parquet-tools-1.12.0.jar dump ./part-00000-2c6c89b1-d165-4528-9a9d-796baa07908e-c000.snappy.parquet
row group 0
--------------------------------------------------------------------------------
ts:  INT64 SNAPPY DO:0 FPO:4 SZ:94/90/0.96 VC:8 ENC:BIT_PACKED,RLE,PLA [more]...

    ts TV=8 RL=0 DL=1 DS: 1 DE:PLAIN_DICTIONARY
    ----------------------------------------------------------------------------
    page 0:                DLE:RLE RLE:BIT_PACKED VLE:PLAIN_DICTIONARY [more]... VC:8

INT64 ts
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 8 ***
value 1: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 2: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 3: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 4: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 5: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 6: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 7: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
value 8: R:0 D:1 V:1001-01-06T22:02:03.123000+0000
{code}
Bug: Filtering is supported only on partition keys of type string Issue
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 06/May/20 05:58
Affects Version/s: 2.4.4

When I submit a SQL with partition filter I see the below error. I tried setting Spark Configuration spark.sql.hive.manageFilesourcePartitions to false but I still see the same issue.

java.lang.RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive.

java.lang.reflect.InvocationTargetException: org.apache.hadoop.hive.metastore.api.MetaException: Filtering is supported only on partition keys of type string

org.apache.hadoop.hive.metastore.api.MetaException: Filtering is supported only on partition keys of type string
Bug: Fix test flakiness caused by MiniKdc which throws "address in use" BindException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/20 16:04
Affects Version/s: 3.0.0


{code:java}
[info] org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite *** ABORTED *** (15 seconds, 426 milliseconds)
[info]   java.net.BindException: Address already in use
[info]   at sun.nio.ch.Net.bind0(Native Method)
[info]   at sun.nio.ch.Net.bind(Net.java:433)
[info]   at sun.nio.ch.Net.bind(Net.java:425)
[info]   at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
[info]   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
[info]   at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
[info]   at org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
[info]   at org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
[info]   at org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
[info]   at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
[info]   at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:748)
{code}


This is an issue fixed in hadoop 2.8.0
https://issues.apache.org/jira/browse/HADOOP-12656

We may need apply the approach from HBASE first before we drop Hadoop 2.7.x

https://issues.apache.org/jira/browse/HBASE-14734
Bug: SQL UI doesn't show metrics and whole stage codegen in AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/20 07:35
Affects Version/s: 3.0.0

When enabling AQE with subqueris within a query, the SQL UI may doesn't show metrics and whole stage codegen.

Here's a reproduce demo:

 
{code:java}
spark.range(10000).toDF("value").write.parquet("/tmp/p1")

spark.range(10000).toDF("value").write.parquet("/tmp/p2")

spark.read.parquet("/tmp/p1").createOrReplaceTempView("t1")

spark.read.parquet("/tmp/p2").createOrReplaceTempView("t2")


spark.sql("select * from t1 where value=(select Max(value) from t2)").show()
{code}
Bug: Improve ExecutorMonitor shuffle-tracking to handle speculative tasks properly
Status: Open | Priority: Major | Resolution: 
Created: 05/May/20 19:43
Affects Version/s: 3.1.0

- [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala#L293-L305]

 
{code:java}
// If the task succeeded and the stage generates shuffle data, record that this executor
// holds data for the shuffle. This code will track all executors that generate shuffle
// for the stage, even if speculative tasks generate duplicate shuffle data and end up
// being ignored by the map output tracker.
//
// This means that an executor may be marked as having shuffle data, and thus prevented
// from being removed, even though the data may not be used.
if (shuffleTrackingEnabled && event.reason == Success) {
  stageToShuffleID.get(event.stageId).foreach { shuffleId =>
    exec.addShuffle(shuffleId)
  }
}
{code}
Bug: Incorrect days conversion by JSON legacy parser
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/20 08:57
Affects Version/s: 3.0.0

Spark 2.4.5:
{code:scala}
scala> val ds = Seq("{'d': '-141704'}").toDS
ds: org.apache.spark.sql.Dataset[String] = [value: string]

scala> val json = spark.read.schema("d date").json(ds)
json: org.apache.spark.sql.DataFrame = [d: date]

scala> json.show
+----------+
|         d|
+----------+
|1582-01-01|
+----------+
{code}

Spark 3.1.0-SNAPSHOT:
{code:scala}
scala> val ds = Seq("{'d': '-141704'}").toDS
ds: org.apache.spark.sql.Dataset[String] = [value: string]

scala> val json = spark.read.schema("d date").json(ds)
json: org.apache.spark.sql.DataFrame = [d: date]

scala> json.show
+----------+
|         d|
+----------+
|1582-01-11|
+----------+
{code}
Bug: Skip timestamp rebasing after 1900-01-01
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/20 12:14
Affects Version/s: 3.0.0

The conversions of Catalyst's DATE/TIMESTAMPS to/from Java's types java.sql.Date/java.sql.Timestamps have almost the same implementation except addition rebasing op. If we look at switch and diffs arrays of all available time zones, we can detect that there is a time point when all diffs are 0. This is 1900-01-01 00:00:00Z. So, we can compare input micros with the time point and skip conversion for modern timestamps.
Bug: Spark Master UI Fails to load if application is waiting for workers to launch driver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/20 11:03
Affects Version/s: 3.0.0

*Steps to Reproduce:*
 # Run Spark Master and 1 Worker (Using local machine).
 # Submit an (long-running) application using spark-submit command: ./bin/spark-submit --master spark://127.0.0.1:7077  --deploy-mode cluster --class org.apache.spark.examples.SparkPi examples/target/original-spark-examples_2.12-3.1.0-SNAPSHOT.jar
 # Load Spark Master UI ([http://localhost:8080/]) which should launch successfully.
 # Submit one more application using the spark-submit command.
 # Load Spark Master UI again.

*Expected Behavior:* Spark Master UI should launch successfully.

*Actual Behavior*: Spark Master UI launch fails with HTTP Error 500 as shown in the below screenshot.

!SparkMasterError.png|width=530,height=411!
Bug: Benchmark rebasing of INT96 and TIMESTAMP_MILLIS timestamps in read/write
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/20 14:34
Affects Version/s: 3.1.0

Add benchmark cases to DateTimeRebaseBenchmark for:
# Read/Write INT96 timestamps
# Read/Write TIMESTAMP_MILLIS w/ rebasing on/off
Bug: Upgrade SLF4J from 1.7.16 to 1.7.30
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/20 04:43
Affects Version/s: 3.0.0

SLF4J 1.7.23+ is required to enable `slf4j-log4j12` with MDC feature to run under Java 9. Also, this will bring all latest bug fixes.
 - [http://www.slf4j.org/news.html]

{quote}When running under Java 9, log4j version 1.2.x is unable to correctly parse the "java.version" system property. Assuming an inccorect Java version, it proceeded to disable its MDC functionality. The slf4j-log4j12 module shipping in this release fixes the issue by tweaking MDC internals by reflection, allowing log4j to run under Java 9. See also SLF4J-393.
{quote}
Bug: Test-jar in the Spark distribution
Status: Open | Priority: Minor | Resolution: 
Created: 01/May/20 11:17
Affects Version/s: 3.0.0

The jar with classifier *tests* is delivered in the Spark distribution:
{code:java}
ls -1 spark-3.0.0-preview2-bin-hadoop2.7/jars/ | grep tests
spark-tags_2.12-3.0.0-preview2-tests.jar
{code}
Normally, test-jars should not be used for production.
Bug: Dataset.joinWith supports JoinType object as input parameter
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 07/May/20 17:52
Affects Version/s: 2.4.5

Current Dataset.joinWith API accepts String type joinType, it doesn't support JoinType object.
I prefer JoinType object (like enum) than String, less chance to have typo and has better readability
{code:scala}
def joinWith[U](other: Dataset[U], condition: Column, joinType: String): Dataset[(T, U)] = {{code}
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala


If I pass LeftOuter.sql to joinType, it will throw exception, since there's a white space in LeftOuter.sql
{code:scala}
case object LeftOuter extends JoinType {
  override def sql: String = "LEFT OUTER"
}
{code}
While the constructor of JoinType only removes underscore, doesn't handle white spaces, 
{code:scala}
object JoinType {
  def apply(typ: String): JoinType = typ.toLowerCase(Locale.ROOT).replace("_", "") match {
    case "inner" => Inner
    case "outer" | "full" | "fullouter" => FullOuter
    case "leftouter" | "left" => LeftOuter
    case "rightouter" | "right" => RightOuter
    case "leftsemi" | "semi" => LeftSemi
    case "leftanti" | "anti" => LeftAnti
    case "cross" => Cross
    case _ =>
      val supported = Seq(
        "inner",
        "outer", "full", "fullouter", "full_outer",
        "leftouter", "left", "left_outer",
        "rightouter", "right", "right_outer",
        "leftsemi", "left_semi", "semi",
        "leftanti", "left_anti", "anti",
        "cross")

      throw new IllegalArgumentException(s"Unsupported join type '$typ'. " +
        "Supported join types include: " + supported.mkString("'", "', '", "'") + ".")
  }
}{code}
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala

I suggest we either add another set of APIs which provide JoinType instead of String, or change JoinType.apply to remove white space as well.
Bug: CSV Writer writes no header for empty DataFrames
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 07/May/20 10:37
Affects Version/s: 2.4.1

When writing a DataFrame as csv with the Header option set to true,
the header is not written when the DataFrame is empty.

This creates failures for processes that read the csv back.

Example (please notice the limit(0) in the second example):
```

 
{code:java}
Welcome to
 ____ __
 / __/__ ___ _____/ /__
 _\ \/ _ \/ _ `/ __/ '_/
 /__ / .__/\_,_/_/ /_/\_\ version 2.4.1
 /_/
Using Python version 2.7.17 (default, Nov 7 2019 10:07:09)
SparkSession available as 'spark'.
>>> df1 = spark.sql("SELECT 1 as a")
>>> df1.limit(1).write.mode("OVERWRITE").option("Header", True).csv("data/test/csv")
>>> spark.read.option("Header", True).csv("data/test/csv").show()
+---+
| a|
+---+
| 1|
+---+
>>> 
>>> df1.limit(0).write.mode("OVERWRITE").option("Header", True).csv("data/test/csv")
>>> spark.read.option("Header", True).csv("data/test/csv").show()
++
||
++
++
{code}
 


Expected behavior:
{code:java}
>>> df1.limit(0).write.mode("OVERWRITE").option("Header", True).csv("data/test/csv")
>>> spark.read.option("Header", True).csv("data/test/csv").show()
+---+
| a|
+---+
+---+{code}
Bug: Null in Date conversion from yyMMddHHmmss for specific date and time
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 25/May/20 08:36
Affects Version/s: 2.3.0

    Hi,

 

We are trying to convert a column with string datatype to date type using below example. It seems to work for all timestamps except for the timestamp for 31st March 2019 02:**:** like  190331020000. Can you please look into it. Thanks.

{code}
scala> sql("select to_date('190331000000','yyMMddHHmmss')").show(false)

+------------------------------------------------+
|         to_date('190331000000', 'yyMMddHHmmss')|
+------------------------------------------------+
|2019-03-31                                      |
+------------------------------------------------+
{code}

 

 

 Interstingly below is not working for highlighted hours (02). 

{code}

scala> sql("select to_date('190331020000','yyMMddHHmmss')").show(false)

+------------------------------------------------+
|           to_date('1903310000', 'yyMMddHHmmss')|
+------------------------------------------------+
|null                                            |
+------------------------------------------------+
{code}
Bug: Font style of Spark SQL DAG-viz is broken in Chrome on macOS
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 02/May/20 20:06
Affects Version/s: 3.1.0

If all the following condition is true, font style of Spark SQL DAG-viz can be broken.
 More specifically, plan name will not be bold style if the plan is WholeStageCodegen.
 * macOS
 * Chrome (version 81)

The current master uses Bootstrap4, which defines the default font family as follows.
{code:java}
-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"
{code}
If we use Chrome, BlinkMacSystemFont is used but font-weight property doesn't work with the font when the font is used in SVG tags.
 This issue is reported here [here|https://bugs.chromium.org/p/chromium/issues/detail?id=1057654]
Bug: Missing sql function, "replace".
Status: Resolved | Priority: Minor | Resolution: Later
Created: 02/May/20 21:19
Affects Version/s: 2.4.4

In the spark sql functions docs, there is a function called [replace|[https://spark.apache.org/docs/latest/api/sql/index.html#replace]]; a non-regex string replace function that should have existed since 2.3.0.

In the spark-shell and in Scala Spark apps (2.4.4) I cannot find this function inside of the `org.apache.spark.sql.functions` package, and it does not appear to be defined in this file for these versions:

[https://github.com/apache/spark/blob/v2.3.0/sql/core/src/main/scala/org/apache/spark/sql/functions.scala]

[https://github.com/apache/spark/blob/v2.4.4/sql/core/src/main/scala/org/apache/spark/sql/functions.scala]

 

However, I can find it in the function registry:

[https://github.com/apache/spark/blob/v2.3.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L338]

[https://github.com/apache/spark/blob/v2.4.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L337]

 

It's not a huge deal because there is still "regexp_replace", it's just a little misleading.
Bug: cast scenarios may generate different results between  Hive and Spark
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/May/20 03:00
Affects Version/s: 2.4.5

`CAST(n,TIMESTAMPTYPE)` If n is Byte/Short/Int/Long data type, Hive treat n as milliseconds unit , while Spark SQL as seconds unit. so the cast result is different,please be care when you use it.

For example:
{code:java}
In spark
spark-sql> select cast(1586318188000 as timestamp);
52238-06-04 13:06:400.0
spark-sql> select cast(1586318188 as timestamp);
2020-04-08 11:56:28

In Hive
hive> select cast(1586318188000 as timestamp);
2020-04-08 11:56:28

hive> select cast(1586318188 as timestamp);
1970-01-19 16:38:38.188{code}
Bug: Upgrade HtmlUnit >= 2.37.0
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/May/20 02:50
Affects Version/s: 3.1.0

Recently, a security issue which affects HtmlUnit is reported.

[https://nvd.nist.gov/vuln/detail/CVE-2020-5529]

According to the report, arbitrary code can be run by malicious users.

HtmlUnit is used for test so the impact might not be large but it's better to upgrade it just in case.
Bug: Backport pyspark Interaction to Spark 2.4.x
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 12/May/20 19:25
Affects Version/s: 2.4.5

In our company, we could really make use of the Interaction pyspark wrapper on spark 2.4.x.

"Interaction" is available on spark 3.0, so I'm proposing to backport the following code to the current Spark2.4.6-rc1:
- https://issues.apache.org/jira/browse/SPARK-26970
- [https://github.com/apache/spark/pull/24426/files]

 

I'm available to pick this up if it's approved.
Bug: Make struct function's output name and class name pretty
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/May/20 03:55
Affects Version/s: 3.0.0

{code}
scala> sql("select struct(1)").show()
+---------------------+
|named_struct(col1, 1)|
+---------------------+
|                  [1]|
+---------------------+
{code}

It should be:

{code}
scala> sql("select struct(1)").show()
+---------+
|struct(1)|
+---------+
|      [1]|
+---------+
{code}
Bug: Remove ResourceInformation in pyspark module's namespace
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/May/20 04:08
Affects Version/s: 3.0.0

Currently we're allowing:

{code}
>>> import pyspark
>>> pyspark.ResourceInformation
<class 'pyspark.resource.information.ResourceInformation'>
{code}

This isn't necessary because resource is now a separate module
Bug: Bump up the minimum Arrow version as 0.15.1 in SparkR
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/May/20 11:36
Affects Version/s: 3.0.0

PySpark side bumped up the minimum Arrow version at SPARK-29376. We should better bump up the version in SparkR side to match with 0.15.1. There's no backward compatibility concern because Arrow optimization in SparkR is new.
Bug: Make spark.files available in driver with cluster deploy mode on kubernetes
Status: Open | Priority: Minor | Resolution: 
Created: 15/May/20 19:21
Affects Version/s: 3.0.0

currently on yarn with cluster deploy mode --files makes the files available for driver and executors and also put them on classpath for driver and executors.

on k8s with cluster deploy mode --files makes the files available on executors but they are not on classpath. it does not make the files available on driver and they are not on driver classpath.

it would be nice if the k8s behavior was consistent with yarn, or at least makes the files available on driver. once the files are available there is a simple workaround to get them on classpath using spark.driver.extraClassPath="./"

background:

we recently started testing kubernetes for spark. our main platform is yarn on which we use client deploy mode. our first experience was that client deploy mode was difficult to use on k8s (we dont launch from inside a pod). so we switched to cluster deploy mode, which seems to behave well on k8s. but then we realized that our program rely on reading files on classpath (application.conf, log4j.properties etc.) that are on the client but now are no longer on the driver (since driver is no longer on client). an easy fix for this seems to be to ship the files using --files to make them available on driver, but we could not get this to work.
Bug: ShuffleBlockFetchIterator keeps localBlocks in its memory even though it never uses it
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 12/May/20 17:52
Affects Version/s: 2.4.5

The {{localBlocks}} is created and used in the {{initialize}} method of ShuffleBlockFetchIterator but is never used after that. 
 It can be local to the initialize method instead of being a field in the {{ShuffleBlockFetchIterator}} instance. It holds on to memory until iterator instance is alive which is unnecessary.
Bug: Register the executor source with the metrics system when running in local mode.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/May/20 13:46
Affects Version/s: 3.0.0

The Apache Spark metrics system provides many useful insights on the Spark workload. In particular, the executor source metrics (https://github.com/apache/spark/blob/master/docs/monitoring.md#component-instance--executor) provide detailed info, including the number of active tasks, some I/O metrics, and task metrics details. Executor source metrics, contrary to other sources (for example ExecutorMetrics source), are not yet available when running in local mode.

This JIRA proposes to register the executor source with the Spark metrics system when running in local mode, as this can be very useful when testing and troubleshooting Spark workloads.
Bug: Add missing keywords in the SQL documents
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/May/20 03:02
Affects Version/s: 3.0.1

Some keywords are missing in the SQL documents and a list of them is as follows. 
 [https://github.com/apache/spark/pull/28290#issuecomment-619321301]
{code:java}
AFTER
CASE/ELSE
WHEN/THEN
IGNORE NULLS
LATERAL VIEW (OUTER)?
MAP KEYS TERMINATED BY
NULL DEFINED AS
LINES TERMINATED BY
ESCAPED BY
COLLECTION ITEMS TERMINATED BY
EXPLAIN LOGICAL
PIVOT
{code}
They should be documented there, too.