Repository: horovod/horovod

Issue:
Estimator failed to save model larger than 2G
**Bug report:**
Please describe erroneous behavior you're observing and steps to reproduce it.

Got error when save large models in metadata file.
```
File "/usr/lib/python3.6/site-packages/horovod/spark/torch/estimator.py", line 62, in saveImpl
param_serializer_fn=_torch_param_serialize)
File "/usr/lib/python3.6/site-packages/horovod/spark/common/serialization.py", line 34, in saveMetadata
sc.parallelize([metadata_json], 1).saveAsTextFile(metadata_path)
File "/ml-code/bazel-bin/platforms/uber_eats/homefeed_conversion_model/path_gen.runfiles/pypi_deps_data_michelangelo/pypi_deps_data_michelangelo_pypi__pyspark_2_4_3/pyspark/context.py", line 527, in parallelize
jrdd = self._serialize_to_jvm(c, serializer, reader_func, createRDDServer)
File "/ml-code/bazel-bin/platforms/uber_eats/homefeed_conversion_model/path_gen.runfiles/pypi_deps_data_michelangelo/pypi_deps_data_michelangelo_pypi__pyspark_2_4_3/pyspark/context.py", line 559, in _serialize_to_jvm
serializer.dump_stream(data, tempFile)
File "/ml-code/bazel-bin/platforms/uber_eats/homefeed_conversion_model/path_gen.runfiles/pypi_deps_data_michelangelo/pypi_deps_data_michelangelo_pypi__pyspark_2_4_3/pyspark/serializers.py", line 345, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
File "/ml-code/bazel-bin/platforms/uber_eats/homefeed_conversion_model/path_gen.runfiles/pypi_deps_data_michelangelo/pypi_deps_data_michelangelo_pypi__pyspark_2_4_3/pyspark/serializers.py", line 142, in dump_stream
self._write_with_length(obj, stream)
File "/ml-code/bazel-bin/platforms/uber_eats/homefeed_conversion_model/path_gen.runfiles/pypi_deps_data_michelangelo/pypi_deps_data_michelangelo_pypi__pyspark_2_4_3/pyspark/serializers.py", line 156, in _write_with_length
raise ValueError("can not serialize object larger than 2G")
ValueError: can not serialize object larger than 2G
```

The culprit is likely to be:

File "/usr/lib/python3.6/site-packages/horovod/spark/common/serialization.py", line 34, in saveMetadata
sc.parallelize([metadata_json], 1).saveAsTextFile(metadata_path)

If metadata_json is >2G, it will hit the Spark limit for serialization. We can use Hadoop API to write it directly.

Here is a Java example for using Hadoop FileSystem API https://stackoverflow.com/questions/27023766/spark-iterate-hdfs-directory




Discussion:

Resolving patch:
diff --git a/horovod/spark/common/serialization.py b/horovod/spark/common/serialization.py
--- a/horovod/spark/common/serialization.py
+++ b/horovod/spark/common/serialization.py
@@ -31,7 +31,12 @@ def saveMetadata(instance, path, sc, extraMetadata=None, paramMap=None,
                                   extraMetadata,
                                   paramMap,
                                   param_serializer_fn)
-        sc.parallelize([metadata_json], 1).saveAsTextFile(metadata_path)
+
+        # save mode by store as spark has 2G limitation on file size.
+        if hasattr(instance, 'getStore') and instance.getStore() is not None:
+            instance.getStore().write_text(metadata_path, metadata_json)
+        else:
+            sc.parallelize([metadata_json], 1).saveAsTextFile(metadata_path)
 
     @staticmethod
     def _get_metadata_to_save(instance, sc, extra_metadata=None, param_map=None,
diff --git a/horovod/spark/common/store.py b/horovod/spark/common/store.py
--- a/horovod/spark/common/store.py
+++ b/horovod/spark/common/store.py
@@ -106,6 +106,10 @@ def read(self, path):
         """Returns the contents of the path as bytes."""
         raise NotImplementedError()
 
+    def write_text(self, path, text):
+        """Write text file to path."""
+        raise NotImplementedError()
+
     def get_local_output_dir_fn(self, run_id):
         raise NotImplementedError()
 
@@ -184,6 +188,10 @@ def read_serialized_keras_model(self, ckpt_path, model, custom_objects):
         model_bytes = self.read(ckpt_path)
         return codec.dumps_base64(model_bytes)
 
+    def write_text(self, path, text):
+        with self.get_filesystem().open(self.get_localized_path(path), 'w') as f:
+            f.write(text)
+
     def is_parquet_dataset(self, path):
         try:
             dataset = self.get_parquet_dataset(path)


Test patch: