av9ash/GitBugs huggingface dataset:
Bug: Removes outer join if it only has DISTINCT on streamed side with alias
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 13:28
Affects Version/s: 3.3.0

{code:scala}
    spark.range(200L).selectExpr("id AS a").createTempView("t1")
    spark.range(300L).selectExpr("id AS b").createTempView("t2")
    spark.sql("SELECT DISTINCT a AS newAlias FROM t1 LEFT JOIN t2 ON a = b").explain(true)
{code}

Expected optimized plan:
{noformat}
== Optimized Logical Plan ==
Aggregate [newAlias#8L], [newAlias#8L]
+- Project [id#0L AS newAlias#8L]
   +- Range (0, 200, step=1, splits=Some(2))
{noformat}
Bug: DynamicJoinSelection.shouldDemoteBroadcastHashJoin should not apply to outer joins
Status: Resolved | Priority: Major | Resolution: Abandoned
Created: 02/Nov/21 03:33
Affects Version/s: 3.1.1

{{DynamicJoinSelection.shouldDemoteBroadcastHashJoin}} will prevent AQE from converting Sort merge join into a broadcast join because SMJ is faster when the side that would be broadcast has a lot of empty partitions.
 This makes sense for inner joins which can short circuit if one side is empty.
 For (left,right) outer join, the streaming side still has to be processed so demoting broadcast join doesn't have the same advantage.
Bug: Print the parsing time for each phase of a SQL
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Nov/21 04:36
Affects Version/s: 2.4.0

the time spent for each phase of a SQL is counted and recorded in QueryPlanningTracker ,  But it doesn't show up anywhere.

when SQL parsing is suspected to be slow, we cannot confirm which phase is slow，therefore, it is necessary to print out the SQL parsing time.
Bug: Make the Java version installed on the container image used by the K8s integration tests with SBT configurable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/21 08:00
Affects Version/s: 3.2.0

I noticed that the default Java version installed on the container image used by the K8s integration tests are different depending on the way to run the tests.

If the tests are launched by Maven, the Java version is 8 is installed.
On the other hand, if the tests are launched by SBT, the Java version is 11.
Further, we have no way to change the version.
Bug: Make FallbackStorageSuite robust in terms of DNS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/21 03:16
Affects Version/s: 3.2.0

Usually, the test case expects the hostname doesn't exist.
{code}
$ ping remote
ping: cannot resolve remote: Unknown host
{code}

In some DNS environments, it returns always.
{code}
$ ping remote
PING remote (23.217.138.110): 56 data bytes
64 bytes from 23.217.138.110: icmp_seq=0 ttl=57 time=8.660 ms
{code}
Bug: Incorrect scope when using named_windows in CTEs
Status: Open | Priority: Major | Resolution: 
Created: 30/Nov/21 08:22
Affects Version/s: 3.1.2

This works, but shouldn't. The named_window is described outside the CTE that uses it.


{code:sql}
with step_1 as (
  select *
  , min(a) over w1 as min_a_over_w1
  from (select 1 as a, 2 as b, 3 as c)
)
select *
from step_1
window w1 as (partition by b order by c)
{code}
Bug: SPARK-33832 brings the bug that OptimizeSkewedJoin may not work since it was applied on whole plan innstead of new stage plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/21 06:26
Affects Version/s: 3.3.0

Since OptimizeSkewedJoin was moved from queryStageOptimizerRules to 

queryStagePreparationRules, the position OptimizeSkewedJoin was applied has been moved from newQueryStage() to reOptimize(). The plan OptimizeSkewedJoin applied on changed from plan of new stage which is about to submit to whole spark plan.

In the cases where skewedJoin is not last stage, OptimizeSkewedJoin may not work because the number of collected shuffleStages is more than 2.

The following test will prove it:

 

 
{code:java}
test("OptimizeSkewJoin may not work") {
  withSQLConf(
    SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
    SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
    SQLConf.SKEW_JOIN_SKEWED_PARTITION_THRESHOLD.key -> "100",
    SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.key -> "100",
    SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM.key -> "1",
    SQLConf.SHUFFLE_PARTITIONS.key -> "10") {
    withTempView("skewData1", "skewData2", "skewData3") {
      spark
        .range(0, 1000, 1, 10)
        .selectExpr("id % 3 as key1", "id % 3 as value1")
        .createOrReplaceTempView("skewData1")
      spark
        .range(0, 1000, 1, 10)
        .selectExpr("id % 1 as key2", "id as value2")
        .createOrReplaceTempView("skewData2")
      spark
        .range(0, 1000, 1, 10)
        .selectExpr("id % 1 as key3", "id as value3")
        .createOrReplaceTempView("skewData3")

      // Query has two skewedJoin in two continuous stages.
      val (_, adaptive1) =
        runAdaptiveAndVerifyResult(
          """
            |SELECT key1 FROM skewData1 s1
            |JOIN skewData2 s2
            |ON s1.key1 = s2.key2
            |JOIN skewData3
            |ON s1.value1 = value3
            |""".stripMargin)
      val shuffles1 = collect(adaptive1) {
        case s: ShuffleExchangeExec => s
      }
      assert(shuffles1.size == 4)
      val smj1 = findTopLevelSortMergeJoin(adaptive1)
      assert(smj1.size == 2 && smj1.forall(_.isSkewJoin))
    }
  }
} {code}
I'll open a PR shortly to fix this issue
Bug: Upgrade kubernetes-client to 5.12.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/21 14:14
Affects Version/s: 3.3.0

We need to add a volcano in [1].

There were some comments on k8s-client repo to tell us how to add support of extension [2]. 

 

And then bump the k8s-client to latest version.

[1] [https://github.com/fabric8io/kubernetes-client/tree/master/extensions]

[2] [https://github.com/fabric8io/kubernetes-client/issues/2565#issuecomment-718150266]
Bug: Backport update pyspark.since annotation to 3.1 and 3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 10:08
Affects Version/s: 3.1.0

Annotation for {{pyspark.since}} has been changed in SPARK-36906, from

{code:python}
def since(version: str) -> Callable[[T], T]: ...
{code}

to

{code:python}
def since(version: Union[str, float]) -> Callable[[T], T]: ...
{code}

We should backport this change to 3.1 and 3.2 (see for example https://github.com/delta-io/delta/pull/305)

This is technically a bug in the downstream projects that runs mypy checks against pyspark.since. When they use it, for example, with {{pyspark.since(3.2)}}, mypy checks fails; however, this case is legitimate. After this change, the mypy change can pass.
Bug: Check unclosed bracketed comments
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/21 13:30
Affects Version/s: 3.3.0

The SQL below has unclosed bracketed comment.
{code:java}
/*abc*/
select 1 as a
/*

2 as b
/*abc*/
, 3 as c

/**/
;
{code}
But Spark will output:
a
1
PostgreSQL also supports the feature, and output:

{code:java}
SQL 错误 [42601]: Unterminated block comment started at position 47 in SQL /*abc*/         -- block comment
select 1 as a
/*

2 as b
/*abc*/
, 3 as c

/**/
. Expected */ sequence
{code}
Bug: Inline type hints for python/pyspark/mllib/linalg/__init__.py
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/21 12:47
Affects Version/s: 3.3.0

Inline type hints from python/pyspark/mlib/linalg/__init__.pyi to python/pyspark/mllib/linalg/__init__.py
Bug: Reduce weights in GaussianMixtureSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/21 02:24
Affects Version/s: 3.3.0

{code}
$ build/sbt "mllib/test"
...
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1760, Failed 1, Errors 0, Passed 1759, Ignored 7
[error] Failed tests:
[error] 	org.apache.spark.ml.clustering.GaussianMixtureSuite
[error] (mllib / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 625 s (10:25), completed Nov 13, 2021, 6:21:13 PM
{code}
Bug: Add `ExtendedRocksDBTest` and disable RocksDB tests on Apple Silicon
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/21 00:32
Affects Version/s: 3.3.0

Java 17 officially support Apple Silicon

- JEP 391: macOS/AArch64 Port
- https://bugs.openjdk.java.net/browse/JDK-8251280

Oracle Java, Azul Zulu, and Eclipse Temurin Java 17 supports Apple Silicon natively.
{code}
/Users/dongjoon/.jenv/versions/oracle17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/zulu17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/temurin17/bin/java: Mach-O 64-bit executable arm64
{code}

Since RocksDBJNI still doesn't support Apple Silicon natively, the following failures occur on M1.
{code}
$ build/sbt "sql/testOnly *RocksDB* *.StreamingSessionWindowSuite"
...
[info] Run completed in 23 seconds, 281 milliseconds.
[info] Total number of tests run: 32
[info] Suites: completed 2, aborted 2
[info] Tests: succeeded 22, failed 10, canceled 0, ignored 0, pending 0
[info] *** 2 SUITES ABORTED ***
[info] *** 10 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.spark.sql.streaming.StreamingSessionWindowSuite
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreIntegrationSuite
[error] Error during tests:
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBSuite
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreSuite
[error] (sql / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 43 s, completed Nov 10, 2021 4:29:50 PM
{code}

This issue aims to add ExtendedRocksDBTest to disable RocksDB selectively on Apple Silicon.
Bug: Collect LocalSparkContext worker logs in case of test failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/21 17:16
Affects Version/s: 3.3.0

About 50 test suites are using LocalSparkContext by specifying "local-cluster" as the cluster URL. In this case executor logs will be under the worker dir which is a temp directory and as such will be deleted at shutdown (for details see https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70)
So when a test fails and the error was on the executor side the log will be lost. 

This is only for local cluster tests and not for standalone tests where logs will be kept in the "<SPARK_HOME>/work".
Bug: When the value of this parameter is greater than the maximum value of int type, the value will be thrown out of bounds. The document description of this parameter should remind the user of this risk point
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 11/Nov/21 03:22
Affects Version/s: 3.2.0

When the value of this parameter is greater than the maximum value of int type, the value will be thrown out of bounds. The document description of this parameter should remind the user of this risk point
Bug: TaskSchedulerImpl should ignore task finished event if its task was already finished state
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/21 07:18
Affects Version/s: 3.2.0

`TaskSchedulerImpl` handle task finished event at `handleSuccessfulTask` and `handleFailedTask` , but in some case the task was already finished state, which we should ignore task finished event.

Case describe: 
when a executor finished a task of some stage, the driver will receive a StatusUpdate event to handle it. At the same time the driver found the executor heartbeat timed out, so the dirver also need handle ExecutorLost event simultaneously. There was a race condition issues here, which will make TaskSetManager.successful and TaskSetManager.tasksSuccessful wrong result. More detailed description and discussion can be viewed at https://issues.apache.org/jira/browse/SPARK-36575 and https://github.com/apache/spark/pull/33872
Bug: spark-sql support nested bracketed comment
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/21 07:26
Affects Version/s: 3.2.0

{code}
/*
select  
select /* BROADCAST(b) */ 4\\;
*/
select  1
;
{code}
failed in spark-sql -f 

{code}
/usr/share/spark-3.2/bin/spark-sql --verbose  -f test.sql
{code}

{code}
Spark master: yarn, Application Id: application_1632999510150_6968442
/* sielect /* BROADCAST(b) */ 4
Error in query:
mismatched input '4' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 30)

== SQL ==
/* sielect /* BROADCAST(b) */ 4
------------------------------^^^

{code}
Bug: Remove unnecessary object serialization on foreachBatch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/21 03:11
Affects Version/s: 3.3.0

Currently, ForeachBatchSink leverages ExternalRDD with converting RDD[InternalRow] to RDD[T], to provide Dataset[T] to the user function. This adds SerializeFromObject in the plan, which is actually not required.

We can leverage LogicalRDD instead, to remove SerializeFromObject from the plan.
Bug: Show hint if analyzer fails due to ANSI type coercion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/21 16:41
Affects Version/s: 3.3.0

Show hint in the error message if analysis failed only with ANSI type coercion:
{code:java}
To fix the error, you might need to add explicit type casts.
To bypass the error with lenient type coercion rules, set spark.sql.ansi.enabled as false. {code}
Bug: Support cast aware output partitioning and required if it can up cast
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Nov/21 08:37
Affects Version/s: 3.3.0

If a `Cast` is up cast then it should be without any truncating or precision lose or possible runtime failures. So the output partitioning should be same with/without `Cast` if the `Cast` is up cast.

Let's say we have a query:
{code:java}
-- v1: c1 int
-- v2: c2 long

SELECT * FROM v2 JOIN (SELECT c1, count(*) FROM v1 GROUP BY c1) v1 ON v1.c1 = v2.c2
{code}
The executed plan contains three shuffle nodes which looks like:
{code:java}
SortMergeJoin
  Exchange(cast(c1 as bigint))
    HashAggregate
      Exchange(c1)
        Scan v1
  Exchange(c2)
    Scan v2
{code}
We can simplify the plan using two shuffle nodes:
{code:java}
SortMergeJoin
  HashAggregate
    Exchange(c1)
      Scan v1
  Exchange(c2)
    Scan v2
{code}
Bug: Char and Varchar breaks backward compatibility between v3 and v2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/21 10:20
Affects Version/s: 3.1.2

We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like `create table`, `alter table` with these types,  the char(n) or varchar(n) will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema.

A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on.
Bug: Missing functionality in spark.pandas
Status: Open | Priority: Major | Resolution: 
Created: 26/Nov/21 09:56
Affects Version/s: 3.2.0

dear,

 

I am missing the functionality to include local variables in the query method.

```

seg = 'A'

psdf.query("segment == @seg")

 

``` 

 

or 

seg = ['A', 'B']

psdf.query("segment == @seg")

```

 

Furthermore I was wondering whether date-offset functionalty as pd.offsets.MonthEnd will be added to future versions.
Bug: Support ANSI intervals and TimestampNTZ for UnionEstimation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/21 03:10
Affects Version/s: 3.3.0

Currently, UnionEstimation doesn't support ANSI intervals and TimestampNTZ. But I think it can support those types because their underlying types are integer or long, which UnionEstimation can compute stats for.
Bug: With enough resources, the task may still be permanently pending
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Nov/21 13:56
Affects Version/s: 3.0.3

{code:java}
// The online environment is actually hive partition data imported to tidb, the code logic can be simplified as follows
    SparkSession testApp = SparkSession.builder()
        .master("local[*]")
        .appName("test app")
        .enableHiveSupport()
        .getOrCreate();
    Dataset<Row> dataset = testApp.sql("select * from default.test where dt = '20211129'");
    dataset.persist(StorageLevel.MEMORY_AND_DISK());
    dataset.count();
{code}

I have observed that tasks are permanently blocked and reruns can always be reproduced.

Since it is only reproducible online, I use the arthas runtime to see the status of the function entries and returns within the TaskSetManager.
https://gist.github.com/guiyanakuang/431584f191645513552a937d16ae8fbd

NODE_LOCAL level, because the persist function is called, the pendingTasks.forHost has a collection of pending tasks, but it points to the machine where the block of partitioned data is located, and since the only resource spark gets is the driver. In this case, it cannot be scheduled. getAllowedLocalityLevel gives the wrong runlevel, so it cannot be run with TaskLocality.Any

The task pending permanently because the scheduling time is very short and it is too late to raise the runlevel with a timeout.
Bug: Cache LogicalPlan.isStreaming() in a lazy val
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/21 08:03
Affects Version/s: 3.2.0

The default implementation of `LogicalPlan.isStreaming()` calls `children.exists(_.isStreaming)`. This can be expensive for large trees, so as a performance optimization I think we should cache the result in a private lazy val.

This is especially important for programs that programmatically construct huge query plans because that will result in multiple analysis passes (and therefore multiple invocations of rules which call `isStreaming`). For example, this the `isStreaming` check accounts for a significant portion of the time in `DeduplicateRelations` (> 20% in my local tests).
Bug: In the latest version release (Spark.3.2.O) in the Apache Spark documentation, the "O" at the end feels wrong, and it is written as the English letter "O"
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Nov/21 07:57
Affects Version/s: 3.2.0

In the latest version release (Spark.3.2.O) in the Apache Spark documentation, the "O" at the end feels wrong, and it is written as the English letter "O".
Bug: BypassMergeSortShuffleWriter may loss data when disk is missing however catagory is present
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Nov/21 07:07
Affects Version/s: 2.4.0

!image-2021-11-27-15-07-41-516.png|width=715,height=484!

We think it has no data when the segment file not exists when all segment files produced by `BypassMergeSortShuffleWriter` is merging;

However, `file.exists()` may rerurn `false` when then the disk which segment file in on is missing and the root catagory exists; the missing disk only lead `file.exists()` return `false` but no exception. The task will run in pease and with no current segment file written.

The segment data will be ignored  and leading shuffle data loss.
Bug: Support subexpression elimination in lambda functions
Status: Open | Priority: Major | Resolution: 
Created: 26/Nov/21 00:44
Affects Version/s: 3.3.0

https://issues.apache.org/jira/browse/SPARK-37019 will add codegen support for higher order functions. However we can't support subexpression elimination inside of lambda functions because subexpressions are evaluated once per row at the beginning of the codegen. Common expressions inside lambda functions can easily result in performance degradation due to multiple evaluations of the same expression. Subexpression elimination inside of lambda functions needs to be handled specially to be evaluated once per function invocation.
Bug: Add small partition factor for rebalance partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/21 13:46
Affects Version/s: 3.3.0

For example `Rebalance` provide a functionality that split the large reduce partition into smalls. However we have seen many SQL produce small files due to the last partition.

Let's say we have one reduce partition and six map partitions and the blocks are: 
[10, 10, 10, 10, 10, 10]
If the target size is 50, we will get two files with 50 and 10. And it will get worse if there are thousands of reduce partitions.

It should be helpful if we can control the min partition size.
Bug: ANSI mode: Use store assignment rules for resolving function invocation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/21 08:50
Affects Version/s: 3.3.0

Under ANSI mode(spark.sql.ansi.enabled=true), the function invocation of Spark SQL:
- In general, it follows the `Store assignment` rules as storing the input values as the declared parameter type of the SQL functions
- Special rules apply for string literals and untyped NULL. A NULL can be promoted to any other type, while a string literal can be promoted to any simple data type.


{code:sql}
> SET spark.sql.ansi.enabled=true;
-- implicitly cast Int to String type
> SELECT concat('total number: ', 1);
total number: 1
-- implicitly cast Timestamp to Date type
> select datediff(now(), current_date);
0

-- specialrule: implicitly cast String literal to Double type
> SELECT ceil('0.1');
1
-- specialrule: implicitly cast NULL to Date type
> SELECT year(null);
NULL

> CREATE TABLE t(s string);
-- Can't store String column as Numeric types.
> SELECT ceil(s) from t;
Error in query: cannot resolve 'CEIL(spark_catalog.default.t.s)' due to data type mismatch
-- Can't store String column as Date type.
> select year(s) from t;
Error in query: cannot resolve 'year(spark_catalog.default.t.s)' due to data type mismatch
{code}
Bug: org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;; despite of time-based window
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 22/Nov/21 16:11
Affects Version/s: 3.0.1

Initially posted here: [https://stackoverflow.com/questions/70062355/org-apache-spark-sql-analysisexception-non-time-based-windows-are-not-supported]

 

'm doing the window-based sorting for the Spark Structured Streaming:
 
{{val filterWindow: WindowSpec = Window  .partitionBy("key")
  .orderBy($"time")

controlDataFrame=controlDataFrame.withColumn("Make Coffee", $"value").    
  withColumn("datetime", date_trunc("second", current_timestamp())).
  withColumn("time", current_timestamp()).
  withColumn("temp_rank", rank().over(filterWindow))
  .filter(col("temp_rank") === 1)
  .drop("temp_rank").
  withColumn("digitalTwinId", lit(digitalTwinId)).
  withWatermark("datetime", "10 seconds")}}

I'm obtaining {{time}} as {{current_timestamp()}} and in schemat I see its type as {{StructField(time,TimestampType,true)}}

Why Spark 3.0 doesn't allow me to do the window operation based on it with the following exception, as the filed is clearly time-based?
 
{{21/11/22 10:34:03 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.

org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;Window [rank(time#163) windowspecdefinition(key#150, time#163 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS temp_rank#171], [key#150], [time#163 ASC NULLS FIRST]
+- Project [key#150, value#151, Make Coffee#154, datetime#158, time#163]}}