av9ash/GitBugs huggingface dataset:
Bug: Eliminate warnings in Avro datasource
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 11/Jan/20 11:54
Affects Version/s: 3.0.0

Here is the list of deprecation warning in Avro:
{code}
/Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala
    Warning:Warning:line (98)value ignoreExtension in class AvroOptions is deprecated (since 3.0): Use the general data source option pathGlobFilter for filtering file names
      if (parsedOptions.ignoreExtension || file.filePath.endsWith(".avro")) {
/Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
    Warning:Warning:line (55)value ignoreExtension in class AvroOptions is deprecated (since 3.0): Use the general data source option pathGlobFilter for filtering file names
        inferAvroSchemaFromFiles(files, conf, parsedOptions.ignoreExtension,
/Users/maxim/proj/eliminate-warning/external/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroPartitionReaderFactory.scala
    Warning:Warning:line (64)value ignoreExtension in class AvroOptions is deprecated (since 3.0): Use the general data source option pathGlobFilter for filtering file names
    if (parsedOptions.ignoreExtension || partitionedFile.filePath.endsWith(".avro")) {
{code}
Bug: pyspark.ml.classification.OneVsRestModel shouldn't have setClassifier, setLabelCol and setWeightCol methods
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/20 23:22
Affects Version/s: 3.0.0

These methods don't makes sense in a model, and not present in the Scala counterpart.

Problem introduced with SPARK-29093.
Bug: Fix ignoreMissingFiles/ignoreCorruptFiles in DSv2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/20 14:30
Affects Version/s: 3.0.0

ignoreMissingFiles/ignoreCorruptFiles in DSv2 is wrong comparing to DSv1, as it stop immediately once it finds a missing or corrupt file while in DSv1 it will skip and continue to read next files.
Bug: orderBy in sortBy clause is removed by EliminateSorts
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Jan/20 13:08
Affects Version/s: 2.4.0

OrderBy in sortBy clause will be removed by EliminateSorts.

code to reproduce:
{code:java}
val dataset = Seq( ("a", 1, 4), ("b", 2, 5), ("b2", 2, 2), ("c", 3, 6) ).toDF("a", "b", "c") 
val groupData = dataset.orderBy("b")
val sortData = groupData.sortWithinPartitions("c")
{code}
The content of groupData is:
{code:java}
partition 0: 
    [a,1,4]
partition 1: 
    [b,2,5]
    [b2,2,2]
partition 2: 
    [c,3,6]{code}
The content of sortData is:
{code:java}
partition 0: 
    [a,1,4]
    [b,2,5]
partition 1: 
    [b2,2,2]
    [c,3,6]{code}
 

UT to cover this defect:

In EliminateSortsSuite.scala
{code:java}
test("should not remove orderBy in sortBy clause") {
  val plan = testRelation.orderBy('a.asc).sortBy('b.desc)
  val optimized = Optimize.execute(plan.analyze)
  val correctAnswer = testRelation.orderBy('a.asc).sortBy('b.desc).analyze
  comparePlans(optimized, correctAnswer)
}{code}
 

 
 This test will be failed because sortBy was removed by EliminateSorts.
Bug: accelerator aware scheduling enforce cores as limiting resource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/20 16:48
Affects Version/s: 3.0.0

For the first version of accelerator aware scheduling(SPARK-27495), the SPIP had a condition that we can support dynamic allocation because we were going to have a strict requirement that we don't waste any resources. This means that the number of number of slots each executor has could be calculated from the number of cores and task cpus just as is done today.

Somewhere along the line of development we relaxed that and only warn when we are wasting resources. This breaks the dynamic allocation logic if the limiting resource is no longer the cores.  This means we will request less executors then we really need to run everything.

We have to enforce that cores is always the limiting resource so we should throw if its not.

I guess we could only make this a requirement with dynamic allocation on, but to make the behavior consistent I would say we just require it across the board.
Bug: SPARK-29976 calculation of slots wrong for Standalone Mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/20 18:49
Affects Version/s: 3.0.0

In SPARK-29976 we added a config to determine if we should allow speculation when the number of tasks is less then the number of slots on a single executor.  The problem is that for standalone mode (and  mesos coarse grained) the EXECUTOR_CORES config is not set properly by default. In those modes the number of executor cores is all the cores of the Worker.    The default of EXECUTOR_CORES is 1.

The calculation:

{color:#000080}val {color}{color:#660e7a}speculationTasksLessEqToSlots {color}= {color:#660e7a}numTasks {color}<= ({color:#660e7a}conf{color}.get({color:#660e7a}EXECUTOR_CORES{color}) / sched.{color:#660e7a}CPUS_PER_TASK{color})

If someone set the cpus per task > 1 then this would end up being false even if 1 task.  Note that the default case where cpus per task is 1 and executor cores is 1 it works out ok but is only applied if 1 task vs number of slots on the executor.

Here we really don't know the number of executor cores for standalone mode or mesos so I think a decent solution is to just use 1 in those cases and document the difference.

Something like max({color:#660e7a}conf{color}.get({color:#660e7a}EXECUTOR_CORES{color}) / sched.{color:#660e7a}CPUS_PER_TASK{color}, 1)
Bug: Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 07/Jan/20 13:58
Affects Version/s: 3.0.0

with accelerator aware scheduling SparkContext.checkResourcesPerTask

Tries to make sure that users have configured things properly and warn or error if not.

It doesn't properly handle all cases like warning if cpu resources are being wasted.  We should test this better and handle those. 

I fixed these in the stage level scheduling but not sure the timeline on getting that in so we may want to fix this separately as well.
Bug: Accelerator aware scheduling handle setting configs to 0 better
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/20 13:52
Affects Version/s: 3.0.0

If you set the resource configs to 0, it errors with divide by zero. While I think ideally the user should just remove the configs we should handle the 0 better.

 

{color:#1d1c1d}$ spark-submit --conf spark.driver.resource.gpu.amount=0 {color}*--conf spark.executor.resource.gpu.amount=0*{color:#1d1c1d} {color}*--conf spark.task.resource.gpu.amount=0*{color:#1d1c1d} --conf spark.driver.resource.gpu.discoveryScript=/shared/tools/get_gpu_resources.sh --conf spark.executor.resource.gpu.discoveryScript=/shared/tools/get_gpu_resources.sh test.py{color}
{color:#1d1c1d}20/01/07 05:36:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable{color}
{color:#1d1c1d}Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties{color}
{color:#1d1c1d}20/01/07 05:36:43 INFO SparkContext: {color}*Running Spark version 3.0.0-preview*
{color:#1d1c1d}20/01/07 05:36:43 INFO ResourceUtils: =============================================================={color}
{color:#1d1c1d}20/01/07 05:36:43 INFO ResourceUtils: Resources for spark.driver:{color}
*gpu -> [name: gpu, addresses: 0]*
{color:#1d1c1d}20/01/07 05:36:43 INFO ResourceUtils: =============================================================={color}
{color:#1d1c1d}20/01/07 05:36:43 INFO SparkContext: Submitted application: test.py{color}
{color:#1d1c1d}......{color}
{color:#1d1c1d}20/01/07 05:36:43 ERROR SparkContext: Error initializing SparkContext.{color}
*java.lang.ArithmeticException: / by zero*
{color:#1d1c1d}at org.apache.spark.SparkContext$.$anonfun$createTaskScheduler$3(SparkContext.scala:2793){color}
{color:#1d1c1d}at org.apache.spark.SparkContext$.$anonfun$createTaskScheduler$3$adapted(SparkContext.scala:2775){color}
{color:#1d1c1d}at scala.collection.Iterator.foreach(Iterator.scala:941){color}
{color:#1d1c1d}at scala.collection.Iterator.foreach$(Iterator.scala:941){color}
{color:#1d1c1d}at scala.collection.AbstractIterator.foreach(Iterator.scala:1429){color}
Bug: Support push down for filters with cast expression
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 02/Jan/20 02:44
Affects Version/s: 3.0.0

Support push down for filters with cast expression, for example:

 
{code:java}
Filter(cast(d, Decimal(9, 3)) = 1000.000)
Filter(cast(d, Boolean) = false)
Filter(cast(d, Int) = 1)
{code}
Bug: FileScan of Data Source V2 doesn't implement Partition Pruning
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 06/Jan/20 03:06
Affects Version/s: 3.0.0

I was trying to understand how Data Source V2 handling partition pruning,  I didn't find the code anywhere which filtering out the unnecessary files in current Data Source V2 implementation. For a File data source, the base class FileScan of Data Source V2 possibly should handle this in "partitions" method. But the current implementation is like the following:

protected def partitions: Seq[FilePartition] = {
 val selectedPartitions = fileIndex.listFiles(Seq.empty, Seq.empty)

 

listFiles passed to empty sequence where no files will be filtered by the partition filter.
Bug: WideSchemaBenchmark fails with OOM
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/20 07:41
Affects Version/s: 3.0.0

Run WideSchemaBenchmark on the master (commit bc16bb1dd095c9e1c8deabf6ac0d528441a81d88) via:
{code}
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark"
{code}
This fails with:
{code}
Caused by: java.lang.reflect.InvocationTargetException
[error] 	at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
[error] 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error] 	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$makeCopy$7(TreeNode.scala:468)
[error] 	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
[error] 	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$makeCopy$1(TreeNode.scala:467)
[error] 	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
[error] 	... 132 more
[error] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[error] 	at java.util.Arrays.copyOfRange(Arrays.java:3664)
[error] 	at java.lang.String.<init>(String.java:207)
[error] 	at java.lang.StringBuilder.toString(StringBuilder.java:407)
[error] 	at org.apache.spark.sql.types.StructType.catalogString(StructType.scala:411)
[error] 	at org.apache.spark.sql.types.StructType.$anonfun$catalogString$1(StructType.scala:410)
[error] 	at org.apache.spark.sql.types.StructType$$Lambda$2441/1040526643.apply(Unknown Source)
{code}
Full stack dump is attached.
Bug: Dropped columns still available for filtering
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 05/Jan/20 11:10
Affects Version/s: 2.4.4

The following minimal example:
{quote}val df = Seq((0, "a"), (1, "b")).toDF("foo", "bar")
df.select("foo").where($"bar" === "a").show
df.drop("bar").where($"bar" === "a").show
{quote}
should result in an error like the following:
{quote}org.apache.spark.sql.AnalysisException: cannot resolve '`bar`' given input columns: [foo];
{quote}
However, it does not but instead works without error, as if the column "bar" would exist.
Bug: Eliminate pypandoc dependency
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jan/20 18:09
Affects Version/s: 3.0.0

PyPI now supports Markdown project descriptions, so we no longer need to convert the Spark README into ReStructuredText and thus no longer need pypandoc.

Removing pypandoc has the added benefit of eliminating the failure mode described in [this PR|https://github.com/apache/spark/pull/18981].
Bug: Change ExpressionEncoder toRow method to return UnsafeRow
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 05/Jan/20 17:45
Affects Version/s: 3.1.0

[~wenchen] observed that the toRow() method on ExpressionEncoder can have its return type specified as UnsafeRow. See discussion on [https://github.com/apache/spark/pull/25024] 

 

Not a high priority but could be done for 3.0.0
Bug: Add public documentation for SPARK_SUBMIT_OPTS
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 17/Jan/20 18:33
Affects Version/s: 2.4.4

Is `SPARK_SUBMIT_OPTS` part of Spark's public interface? If so, it needs some documentation. I cannot see it documented [anywhere|https://github.com/apache/spark/search?q=SPARK_SUBMIT_OPTS&unscoped_q=SPARK_SUBMIT_OPTS] in the docs.

How do you use it? What is it useful for? What's an example usage? etc.
Bug: Why can PySpark persist data only in serialised format?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 20/Jan/20 08:43
Affects Version/s: 2.4.0

The storage levels in PySpark allow to persist data only in serialised format. There is also [a comment|[https://github.com/apache/spark/blob/master/python/pyspark/storagelevel.py#L28]] explicitly stating that "Since the data is always serialized on the Python side, all the constants use the serialized formats." While that makes totally sense for RDDs, it is not clear to me why it is not possible to persist data without serialisation when using the dataframe/dataset APIs. In theory, in such cases, the persist would only be a directive and data would never leave the JVM, thus allowing for un-serialised persistence, correct? Many thanks for the feedback!
Bug: Regressions in Join benchmarks
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jan/20 17:06
Affects Version/s: 3.0.0

Regenerated benchmark results in the https://github.com/apache/spark/pull/27078 shows many regressions in JoinBenchmark. The benchmarked queries slowed down by up to 3 times, see
old results:
https://github.com/apache/spark/pull/27078/files#diff-d5cbaab2b49ee9fddfa0e229de8f607dL10
new results:
https://github.com/apache/spark/pull/27078/files#diff-d5cbaab2b49ee9fddfa0e229de8f607dR10

One of the difference in queries is using the `NoOp` datasource in new queries.
Bug: Avoid rebuilding `AvroOptions` per each partition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jan/20 10:35
Affects Version/s: 3.0.0

Currently, an instance of `AvroOptions` is created per each partition. This can be avoid by building it only once and pass to `AvroScan`. See https://github.com/apache/spark/pull/27174#discussion_r365596481 for more details.
Bug: Return Iterable from FailureSafeParser.rawParser
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jan/20 16:19
Affects Version/s: 2.4.4

Currently, rawParser of FailureSafeParser has the signature `IN => Seq[InternalRow]` which is strict requirement for underlying rawParser. It could return Option, for example like CSV datasource. The ticket aims to refactor FailureSafeParser, and change Seq to Iterable. Also. need to modify CSV and JSON datasources where FailureSafeParser is used, basically.
Bug: Iterator doesn't refer outer identifier named "iterator" properly in Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jan/20 19:18
Affects Version/s: 3.0.0

As of Scala 2.13, scala.collection.Iterator has "iterator" method so if an inner class of Iterator means to refer an outer identifier named "iterator", it does not work as we think.
Following is an example.

{code}
val iterator = ...

return new Iterator {

  def next() {
     iterator.next()         // this "iterator" is not what we defined above.
  }
}
{code}
Bug: Eliminate deprecation warnings for ExpressionInfo
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jan/20 13:40
Affects Version/s: 3.0.0

{code}
Warning:(335, 5) constructor ExpressionInfo in class ExpressionInfo is deprecated: see corresponding Javadoc for more information.
    new ExpressionInfo("noClass", "myDb", "myFunction", "usage", "extended usage"),
Warning:(732, 5) constructor ExpressionInfo in class ExpressionInfo is deprecated: see corresponding Javadoc for more information.
    new ExpressionInfo("noClass", "myDb", "myFunction2", "usage", "extended usage"),
Warning:(751, 5) constructor ExpressionInfo in class ExpressionInfo is deprecated: see corresponding Javadoc for more information.
    new ExpressionInfo("noClass", "myDb", "myFunction2", "usage", "extended usage"),
{code}
Bug: Publicly document options under spark.sql.*
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jan/20 18:37
Affects Version/s: 3.0.0

SPARK-20236 added a new option, {{spark.sql.sources.partitionOverwriteMode}}, but it doesn't appear to be documented in [the expected place|http://spark.apache.org/docs/2.4.4/configuration.html]. In fact, none of the options under {{spark.sql.*}} that are intended for users are documented on spark.apache.org/docs.

We should add a new documentation page for these options.
Bug: Remove 1.x testthat switch once Jenkins version is updated to 2.x
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jan/20 14:28
Affects Version/s: 3.0.0

As part of SPARK-23435 proposal we include {{testthat}} 1.x compatibility mode 

{code}
    if (grepl("^1\\..*", packageVersion("testthat"))) {
                         NULL,	      # testthat 1.x
                         "summary")	      test_runner <- testthat:::run_tests
      reporter <- "summary"

    } else {
      # testthat >= 2.0.0
      test_runner <- testthat:::test_package_dir
      reporter <- testthat::default_reporter()
    }
{code}

in {{R/pkg/tests/run-all.R}}.

It should be removed once whole infrastructure uses {{testhat}} 2.x or later.
Bug: Explicitly note that we don't add "pandas compatible" aliases
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Jan/20 01:43
Affects Version/s: 3.0.0

We added "pandas compatible" aliases as of [https://github.com/apache/spark/commit/c84d91692aa25c01882bcc3f9fd5de3cfa786195] and [https://github.com/apache/spark/commit/b6bf4f76c78abfaafa99b3c3c08b498aa9644346]

There are too many differences and I don't think it makes sense to extend such supports.

This Jira targets to document this.
Bug: Why does SparkSQL allow `WHERE` to be table alias?
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 23/Jan/20 09:27
Affects Version/s: 2.4.4

An empty `WHERE` expression is valid in Spark SQL, as: `SELECT * FROM XXX WHERE`. Here `WHERE` is parsed as the table alias.

I think this surprises most SQL users, as this is an invalid statement in some SQL engines like MySQL.  

I checked the source code, and found more keywords (in most SQL system) are treated as `noReserved` and allowed to be table alias.  

Could anyone please give the rationality behind this decision?
Bug: saveAsTable does not honor spark.hadoop.hive.warehouse.subdir.inherit.perms
Status: Resolved | Priority: Minor | Resolution: Incomplete
Created: 02/Jan/20 16:26
Affects Version/s: 2.4.4

{code}
-bash-4.2$ hdfs dfs -ls /tmp | grep my_databases
 drwxr-x--T - redsanket users 0 2019-12-04 20:15 /tmp/my_databases
{code}

{code}
>>> spark.sql("CREATE TABLE redsanket_db.example(bcookie string, ip int) STORED AS orc");
{code}

{code}
-bash-4.2$ hdfs dfs -ls /tmp/my_databases | grep example
 drwxr-x--T - redsanket users 0 2019-12-04 20:20 /tmp/my_databases/example
{code}

Now after {{saveAsTable}}

{code}
 >>> data = [('First', 1), ('Second', 2), ('Third', 3), ('Fourth', 4), ('Fifth', 5)]
 >>> df = spark.createDataFrame(data)
 >>> df.write.format("orc").mode('overwrite').saveAsTable('redsanket_db.example')
{code}

{code}
-bash-4.2$ hdfs dfs -ls /tmp/my_databases | grep example
 drwx------ - redsanket users 0 2019-12-04 20:23 /tmp/my_databases/example
{code}

 Overwrites the permissions

Insert into honors preserving parent directory permissions.

{code}
 >>> spark.sql("DROP table redsanket_db.example");
 DataFrame[]
 >>> spark.sql("CREATE TABLE redsanket_db.example(bcookie string, ip int) STORED AS orc");
 DataFrame[]
 >>> df.write.format("orc").insertInto('redsanket_db.example')
{code}

{code}
-bash-4.2$ hdfs dfs -ls /tmp/my_databases | grep example
 drwxr-x--T - redsanket users 0 2019-12-04 20:43 /tmp/my_databases/example
{code}

 It is either limitation of the API based on the mode and the behavior has to be documented or needs to be fixed
Bug: Eliminate warnings in Java tests regarding to deprecated API
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Jan/20 17:11
Affects Version/s: 2.4.4

Suppress warnings about deprecated Spark API in Java test suites:
{code}
/Users/maxim/proj/eliminate-warnings-part2/sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetAggregatorSuite.java
    Warning:Warning:line (32)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (91)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (100)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (109)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (118)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
{code}
{code}
/Users/maxim/proj/eliminate-warnings-part2/sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java
    Warning:Warning:line (28)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (37)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (46)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (55)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
    Warning:Warning:line (64)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated
{code}
{code}
/Users/maxim/proj/eliminate-warnings-part2/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java
    Warning:Warning:line (478)java: json(org.apache.spark.api.java.JavaRDD<java.lang.String>) in org.apache.spark.sql.DataFrameReader has been deprecated
{code}
Bug: PeriodicRDDCheckpointer supports storageLevel
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/20 11:32
Affects Version/s: 3.0.0

Intermediate RDDs in ML are cached with storageLevel=StorageLevel.MEMORY_AND_DISK.

PeriodicRDDCheckpointer will store RDD with storageLevel=StorageLevel.MEMORY_ONLY, it maybe nice to set the storageLevel of checkpointer.