av9ash/GitBugs huggingface dataset:
Bug: refreshListener() API on StreamingQueryManager for spark connect
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 14/Aug/23 19:03
Affects Version/s: 4.0.0

I’m thinking of an improvement for connect python listener and foreachBatch. Currently if you define a variable outside of the function, you can’t actually see it on client side if it’s touched in the function because the operation is on server side, e.g. 

 

 
{code:java}
x = 0
class MyListener(StreamingQueryListener):
    def onQueryStarted(e):
        x = 100
        self.y = 200
spark.streams.addListener(MyListener())
q = spark.readStream.....start()
 
# x is still 0, self.y is not defined
{code}
 

 

But for the self.y case, there could be an improvement. 

 

The server side is capable of pickle serialize the whole listener instance again, and send back to the client.

 

So we could define a new interface on the streaming query manager, maybe called refreshListener(listener), e.g. use it as `spark.streams.refreshListener()`

 

 
{code:java}
def refreshListener(listener: StreamingQueryListener) -> StreamingQueryListener
 # send the listener id with this refresh request, server receives this request and serializes the listener again, and send back to client, so the returned new listener contains the updated value self.y
 
{code}
 

For `foreachBatch`, we could wrap the function to a new class
Bug: Do not combine multiple Generate operators in the same WholeStageCodeGen node because it can  easily cause OOM failures if arrays are relatively large
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/23 09:23
Affects Version/s: 3.0.0

This is an issue since the WSCG  implementation of the generate node. 

Because WSCG compute rows in batches , the combination of WSCG and the explode operation consume a lot of the dedicated executor memory. This is even more true when the WSCG node contains multiple explode nodes. This is the case when flattening a nested array.

The generate node used to flatten array generally  produces an amount of output rows that is significantly higher than the input rows.

the number of output rows generated is even drastically higher when flattening a nested array .

When we combine more that 1 generate node in the same WholeStageCodeGen  node, we run  a high risk of running out of memory for multiple reasons. 

1- As you can see from snapshots added in the comments ,  the rows created in the nested loop are saved in a writer buffer.  In this case because the rows were big , the job failed with an Out Of Memory Exception error .

2_ The generated WholeStageCodeGen result in a nested loop that for each row  , will explode the parent array and then explode the inner array.  The rows are accumulated in the writer buffer without accounting for the row size.

Please view the attached Spark Gui and Spark Dag 

In my case the wholestagecodegen includes 2 explode nodes. 

Because the array elements are large , we end up with an Out Of Memory error. 

 

I recommend that we do not merge  multiple explode nodes in the same whole stage code gen node . Doing so leads to potential memory issues.

In our case , the job execution failed with an  OOM error because the the WSCG executed  into a nested for loop .
Bug: Add the `TIMEDIFF` aliases for `TIMESTAMPDIFF()`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Aug/23 17:01
Affects Version/s: 4.0.0

Introduce the timediff() function, which takes three arguments: unit, and two datetime expressions, i.e.,
{code:sql}
datediff(unit, startDatetime, endDatetime)
{code}
The function can be an alias to timestampdiff().
Bug: Improve initialization of SparkConnectService
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/23 18:49
Affects Version/s: 4.0.0

Many SparkConnectService components are vals in a global singleton object. They don't get initialized when SparkConnectService.start() is called, but whenever that object gets first touched. Because of that, some of the fields must have been made lazy val, some are var that are initialized to null and initialized during start().

Clean it up. It would be best if Spark Connect server components did not depend on global singleton, but got properly initialized as an object instance.
Bug: SaveMode.ErrorIfExists does not work with kafka-sql
Status: Open | Priority: Major | Resolution: 
Created: 11/Aug/23 09:46
Affects Version/s: 3.4.1

I' trying to write batch dataframe to Kafka topic with {{mode="error"}}, but when topic exists it does not raise exception. Instead it appends data to a topic.

Steps to reproduce:

1. Start Kafka:

docker-compose.yml
{code:yaml}
version: '3.9'

services:
  zookeeper:
    image: bitnami/zookeeper:3.8
    environment:
      ALLOW_ANONYMOUS_LOGIN: 'yes'

  kafka:
    image: bitnami/kafka:latest
    restart: unless-stopped
    ports:
    - 9093:9093
    environment:
      ALLOW_PLAINTEXT_LISTENER: 'yes'
      KAFKA_ENABLE_KRAFT: 'no'
      KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL_PLAINTEXT_ANONYMOUS
      KAFKA_CFG_LISTENERS: INTERNAL_PLAINTEXT_ANONYMOUS://:9092,EXTERNAL_PLAINTEXT_ANONYMOUS://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL_PLAINTEXT_ANONYMOUS://kafka:9092,EXTERNAL_PLAINTEXT_ANONYMOUS://localhost:9093
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL_PLAINTEXT_ANONYMOUS:PLAINTEXT,EXTERNAL_PLAINTEXT_ANONYMOUS:PLAINTEXT
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: 'true'
    depends_on:
    - zookeeper
{code}

{code:bash}
docker-compose up -d
{code}

2. Start Spark session:

{code:bash}
pip install pyspark[sql]==3.4.1
{code}


{code:python}
from pyspark.sql import SparkSession

spark = SparkSession.builder.config("spark.jars.packages", "org.apache.spark:spark-sql-kafka-0-10_2.12:3.4.1").getOrCreate()
{code}

3. Create DataFrame and write it to Kafka. First write using {{mode="append"}} to create topic, then with {{mode="error"}} to raise because topic already exist:
{code}
df = spark.createDataFrame([{"value": "string"}])
df.write.format("kafka").option("kafka.bootstrap.servers", "localhost:9093").option("topic", "new_topic").mode("append").save()

# no exception is raised
df.write.format("kafka").option("kafka.bootstrap.servers", "localhost:9093").option("topic", "new_topic").mode("error").save()
{code}

4. Check topic content - 2 rows are added to topic instead of one:
{code:python}
spark.read.format("kafka").option("kafka.bootstrap.servers", "localhost:9093").option("subscribe", "new_topic").load().show(10, False)
{code}
{code}
+----+-------------------+---------+---------+------+-----------------------+-------------+
|key |value              |topic    |partition|offset|timestamp              |timestampType|
+----+-------------------+---------+---------+------+-----------------------+-------------+
|null|[73 74 72 69 6E 67]|new_topic|0        |0     |2023-08-11 09:39:35.813|0            |
|null|[73 74 72 69 6E 67]|new_topic|0        |1     |2023-08-11 09:39:36.122|0            |
+----+-------------------+---------+---------+------+-----------------------+-------------+
{code}

It looks like mode is checked by KafkaSourceProvider, but is not used at all:
https://github.com/apache/spark/blob/v3.4.1/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L172-L178

So data is always appended to topic.
Bug: Add a displayOrder variable to WebUITab to specify the order in which tabs appear
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/23 22:55
Affects Version/s: 3.5.1

Add a displayOrder variable to WebUITab to specify the order in which tabs appear. Currently, the tabs are ordered by when they get attached, which isn't always desired. The default is MIN_VALUE, meaning if it's not specified, it will appear in the order added before any tabs with a non-default displayOrder.  For example, we would like to have the SQL Tab appear before the Connect tab; however, based on the code flow, the Connect tab will be attached first and with the current logic, that tab would also appear first.
Bug: Fix a bug of promoting string as double in binary arithmetic with interval  
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/23 18:03
Affects Version/s: 4.0.0

The following query works on branch-3.5 or below, but fails on the latest master:

```
select concat(DATE'2020-12-31', ' ', date_format('09:03:08', 'HH:mm:ss')) + (INTERVAL '03' HOUR)
```
 
The direct reason is now we mark `cast(date as string)` as resolved during type coercion after changes [https://github.com/apache/spark/pull/42089.] As a result, there are two transforms from CombinedTypeCoercionRule
```
Rule ConcatCoercion Transformed concat(2020-12-31,  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles))) to concat(cast(2020-12-31 as string),  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles)))

Rule PromoteStrings Transformed (concat(cast(2020-12-31 as string),  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles))) + INTERVAL '03' HOUR) to (cast(concat(cast(2020-12-31 as string),  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles))) as double) + INTERVAL '03' HOUR)
```  
The second transform doesn't happen in previous releases since cast(2020-12-31 as string)  used to be unresolved after the first transform.
 
The fix is simple, the analyzer should not promote string as double in binary arithmetic with ANSI interval. The changes in [https://github.com/apache/spark/pull/42089|https://github.com/apache/spark/pull/42089.] are valid and we should keep it.
Bug: Local tmp data is not cleared while using spark streaming consuming from kafka
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/23 06:56
Affects Version/s: 3.2.0

we are using spark 3.2 consuming data from kafka and then using `collectAsMap` to send to driver, we found the local temp file do not get cleared if the data consumed from kafka is larger than 200m(spark.network.maxRemoteBlockSizeFetchToMem)

!https://intranetproxy.alipay.com/skylark/lark/0/2023/png/320711/1691419276170-2dd0964f-4cf4-4b15-9fbe-9622116671da.png!
Bug: Improve DeduplicateRelations rewriteAttrs compatibility
Status: Open | Priority: Major | Resolution: 
Created: 10/Aug/23 06:51
Affects Version/s: 3.5.0

{{Follow [https://github.com/apache/spark/pull/41554,] we should add test for }}

{{{}MapPartitionsInR{}}}, {{{}MapPartitionsInRWithArrow{}}}, {{{}MapElements{}}}, {{{}MapGroups{}}}, {{{}FlatMapGroupsWithState{}}}, {{{}FlatMapGroupsInR{}}}, {{{}FlatMapGroupsInR{}}}, {{{}FlatMapGroupsInRWithArrow{}}}, {{{}FlatMapGroupsInPandas{}}}, {{{}MapInPandas{}}}, {{{}PythonMapInArrow{}}}, {{FlatMapGroupsInPandasWithState and}} {{FlatMapCoGroupsInPandas}}

{{To make sure DeduplicateRelations rewriteAttrs will rewrite its attribute normally. Also should fix the error behavior follow [https://github.com/apache/spark/pull/41554|https://github.com/apache/spark/pull/41554,]}}
Bug: Ease restriction of LCA resolution regarding queries with having
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Aug/23 01:19
Affects Version/s: 3.4.2

Current LCA resolution has a limitation, that it can't resolve the query, when it satisfies all the following criteria:
 # the main (outer) query has having clause
 # there is a window expression in the query
 # in the same SELECT list as the window expression in 2), there is an lca

This is because LCA won't rewrite plan until UNRESOLVED_HAVING is resolved; window expressions won't be extracted until LCA in the same SELECT lists are rewritten; however UNRESOLVED_HAVING depends on the child to be resolved, which could include the Window. It becomes a deadlock.

*We should ease some limitation on the LCA resolution regarding to having, to break the deadlock for most cases.*

For example, for the following query:
{code:java}
create table t (col boolean) using orc;
with w AS (
  select min(col) over () as min_alias,
  min_alias as col_alias
  FROM t
)
select col_alias
from w
having count > 0;
{code}

 
It now throws confusing error message:
{code:java}
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `col_alias` 
cannot be resolved. Did you mean one of the following? [`col_alias`, `min_alias`].{code}

The LCA and window is in a CTE that is completely unrelated to the having. LCA should resolve in this case.
Bug: Incorrect limit handling and config parsing in Arrow collect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/23 12:46
Affects Version/s: 3.4.0

In the arrow writer [code|https://github.com/apache/spark/blob/6161bf44f40f8146ea4c115c788fd4eaeb128769/sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala#L154-L163] , the conditions don’t seem to hold what the documentation says regd "{_}maxBatchSize and maxRecordsPerBatch, respect whatever smaller"{_} since it seems to actually respect the conf which is "larger" (i.e less restrictive) due to _||_ operator.

 

Further, when the `{_}CONNECT_GRPC_ARROW_MAX_BATCH_SIZE{_}` conf is read, the value is not converted to bytes from Mib ([example|https://github.com/apache/spark/blob/3e5203c64c06cc8a8560dfa0fb6f52e74589b583/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/SparkConnectPlanExecution.scala#L103]).
Bug: parameter markers are not blocked from DEFAULT (and other places)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Aug/23 16:33
Affects Version/s: 3.4.0

scala> spark.sql("CREATE TABLE t11(c1 int default :parm)", args = Map("parm" -> 5)).show()

-> success

scala> spark.sql("describe t11");

[INVALID_DEFAULT_VALUE.UNRESOLVED_EXPRESSION] Failed to execute EXISTS_DEFAULT command because the destination table column `c1` has a DEFAULT value :parm, which fails to resolve as a valid expression.

This likely extends to other DDL-y places.
I can only find protection against placement in the body of a CREATE VIEW.

I see two ways out of this:
* Raise an error (as we do for CREATE VIEW v1(c1) AS SELECT ? )
 * Improve the way we persist queries/expressions to substitute the at-DDL-time bound parameter value (it' not a bug it's a feature....)
Bug: Log eventLog rewrite duration when compact old event log files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Aug/23 10:55
Affects Version/s: 3.4.1

When enable {{spark.eventLog.rolling.enabled}} and the number of eventLog files exceeds the value of {{{}spark.history.fs.eventLog.rolling.maxFilesToRetain{}}},
HistoryServer will compact the old event log files into one compact file.

Currently there is no log the rewrite duration in {{rewrite}} method, this metric is useful for understand the compact duration, so we need add logs in the method.
Bug: Migrate ‎test_timedelta_ops assert_eq to use assertDataFrameEqual
Status: Open | Priority: Major | Resolution: 
Created: 08/Aug/23 00:45
Affects Version/s: 3.5.0

Migrate assert_eq to assertDataFrameEqual in this file: [‎python/pyspark/pandas/tests/data_type_ops/test_timedelta_ops.py|https://github.com/databricks/runtime/blob/f579860299b16f6614f70c7cf2509cd89816d363/python/pyspark/pandas/tests/data_type_ops/test_timedelta_ops.py#L176]
Bug: Migrate test_series_conversion assert_eq to use assertDataFrameEqual
Status: Open | Priority: Major | Resolution: 
Created: 08/Aug/23 00:07
Affects Version/s: 3.5.0

Migrate assert_eq to assertDataFrameEqual in this file: 
[‎python/pyspark/pandas/tests/test_series_conversion.py|https://github.com/databricks/runtime/blob/d162bd182de8bcea180d874027edb86ae8fc60e5/python/pyspark/pandas/tests/test_series_conversion.py#L63]
Bug: Create table like other table should also copy table stats.
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Aug/23 06:36
Affects Version/s: 3.4.1

For example:
describe table extended tbl;

col0                    int
col1                    int
col2                    int
col3                    int

Detailed Table Information
Catalog                 spark_catalog
Database                default
Table                   tbl
Owner                   zhuqi
Created Time            Mon Aug 07 14:02:30 CST 2023
Last Access             UNKNOWN
Created By              Spark 4.0.0-SNAPSHOT
Type                    MANAGED
Provider                hive
Table Properties        [transient_lastDdlTime=1691388473]
Statistics              30 bytes
Location                [file:/Users/zhuqi/spark/spark/spark-warehouse/tbl|file:///Users/zhuqi/spark/spark/spark-warehouse/tbl]
Serde Library           org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat             org.apache.hadoop.mapred.TextInputFormat
OutputFormat            org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties      [serialization.format=1]
Partition Provider      Catalog
Time taken: 0.032 seconds, Fetched 23 row(s)

create table tbl2 like tbl;
23/08/07 14:14:07 WARN HiveMetaStore: Location: [file:/Users/zhuqi/spark/spark/spark-warehouse/tbl2|file:///Users/zhuqi/spark/spark/spark-warehouse/tbl2] specified for non-external table:tbl2
Time taken: 0.098 seconds
spark-sql (default)> describe table extended tbl2;
col0                    int
col1                    int
col2                    int
col3                    int

Detailed Table Information
Catalog                 spark_catalog
Database                default
Table                   tbl2
Owner                   zhuqi
Created Time            Mon Aug 07 14:14:07 CST 2023
Last Access             UNKNOWN
Created By              Spark 4.0.0-SNAPSHOT
Type                    MANAGED
Provider                hive
Table Properties        [transient_lastDdlTime=1691388847]
Location                [file:/Users/zhuqi/spark/spark/spark-warehouse/tbl2|file:///Users/zhuqi/spark/spark/spark-warehouse/tbl2]
Serde Library           org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat             org.apache.hadoop.mapred.TextInputFormat
OutputFormat            org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties      [serialization.format=1]
Partition Provider      Catalog
Time taken: 0.03 seconds, Fetched 22 row(s)

The table stats are missing.
Bug: Support deal with microsecond for `from_json` function
Status: Open | Priority: Major | Resolution: 
Created: 07/Aug/23 03:47
Affects Version/s: 3.4.1

Currently, `from_json` function will treat all input timestamp as second precise.
However, if the input is millisecond or microsecond, this function will return wrong result and throw no exception, which will lead to misunderstanding.

Example:
{code:java}
spark-sql> SELECT from_json('{"a":1, "b":1691241070}', 'a INT, b TIMESTAMP');
{"a":1,"b":2023-08-05 21:11:10}

spark-sql> SELECT from_json('{"a":1, "b":1691241070000}', 'a INT, b TIMESTAMP');
{"a":1,"b":+55563-04-19 18:06:40}

spark-sql> SELECT from_json('{"a":1, "b":1691241070000000}', 'a INT, b TIMESTAMP');
{"a":1,"b":-183707-06-27 20:24:24.251328}{code}
Bug: Should check if `free_disk_space` exists first, then proceed to execution.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/23 07:36
Affects Version/s: 4.0.0

We added `free_disk_space` and `free_disk_space_container` to clean up the disk, but because the daily tests of other branches and the master branch share the yml file, we should check if the file exists before execution, otherwise it will affect the daily tests of other branches.

 
 * [https://github.com/apache/spark/actions/runs/5761479443]
 * [https://github.com/apache/spark/actions/runs/5760423900]
 * https://github.com/apache/spark/actions/runs/5759384052
Bug: Fix Scala 2.13 mima check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/23 06:20
Affects Version/s: 4.0.0

[https://github.com/apache/spark/actions/runs/5695413124/job/15438535023]

 
{code:java}
[error] spark-core: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered 4013)
8218[error]  * the type hierarchy of object org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages#SparkAppConfig is different in current version. Missing types {scala.runtime.AbstractFunction4}
8219[error]    filter with: ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$SparkAppConfig$")
8220[error] java.lang.RuntimeException: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered 4013)
8221[error] 	at scala.sys.package$.error(package.scala:30)
8222[error] 	at com.typesafe.tools.mima.plugin.SbtMima$.reportModuleErrors(SbtMima.scala:89)
8223[error] 	at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$2(MimaPlugin.scala:36)
8224[error] 	at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$2$adapted(MimaPlugin.scala:26)
8225[error] 	at scala.collection.Iterator.foreach(Iterator.scala:943)
8226[error] 	at scala.collection.Iterator.foreach$(Iterator.scala:943)
8227[error] 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
8228[error] 	at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$1(MimaPlugin.scala:26)
8229[error] 	at com.typesafe.tools.mima.plugin.MimaPlugin$.$anonfun$projectSettings$1$adapted(MimaPlugin.scala:25)
8230[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
8231[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
8232[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
8233[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
8234[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
8235[error] 	at sbt.Execute.work(Execute.scala:292)
8236[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
8237[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
8238[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
8239[error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
8240[error] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
8241[error] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
8242[error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
8243[error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
8244[error] 	at java.lang.Thread.run(Thread.java:750)
8245[error] (core / mimaReportBinaryIssues) Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.4.0! Found 1 potential problems (filtered 4013)
8246[error] Total time: 172 s (02:52), completed Jul 28, 2023 7:26:06 PM {code}
Bug: Runtime min-max filter
Status: Open | Priority: Major | Resolution: 
Created: 05/Aug/23 02:17
Affects Version/s: 3.2.1

We can infer min-max index when building bloom filter and push it down to datasource. 
 # Min-max index can skip part of data before loading them into memory.
 # building min-max index can be done along with bloom filter building, so aggregation for bloom filter building can be reused.
Bug: Improve error messages for creating Python UDTFs with pickling errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/23 22:12
Affects Version/s: 3.5.0

Currently, when users create a Python UDTF with a non-pickleable object, it throws this error:

_pickle.PicklingError: Cannot pickle files that are not opened for reading: w

 

We should make this more user-friendly