Repository: kedro-org/kedro

Issue:
[KED-2140]TensorflowModelDataset save fails with hdf5 model when versioning is enabled.
## Description
The title is pretty self-explanatory.

## Context
By default, `TensorFlowModelDataset` saves a model using TensorFlow's native format. This works as expected. Saving the model as an HDF5 file also works as expected, provided you don't have versioning enabled.

## Steps to Reproduce
This failure happens with a catalog configuration like the following:

```
trained_model:
  type: tensorflow.TensorFlowModelDataset
  filepath: data/06_models/fully_trained.hd5
  save_args:
    save_format: h5
  versioned: True
```

## Expected Result
The model should be saved successfully as a versioned `DataSet`.

## Actual Result
You get an error like the following:

```
Traceback (most recent call last):
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/io/core.py", line 240, in save
    self._save(data)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/extras/datasets/tensorflow/tensorflow_model_dataset.py", line 167, in _save
    self._fs.copy(path, save_path)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/fsspec/implementations/local.py", line 90, in copy
    shutil.copyfile(path1, path2)
  File "/home/daniel/.pyenv/versions/3.7.7/lib/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/home/daniel/git/cotton_counter/data/06_models/fully_trained.hd5/2020-09-19T16.20.54.312Z/fully_trained.hd5'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/daniel/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/daniel/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/__main__.py", line 38, in <module>
    main()
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/framework/cli/cli.py", line 724, in main
    cli_collection()
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/daniel/git/cotton_counter/kedro_cli.py", line 263, in run
    pipeline_name=pipeline,
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/framework/context/context.py", line 767, in run
    raise exc
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/framework/context/context.py", line 759, in run
    run_result = runner.run(filtered_pipeline, catalog, run_id)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/runner/runner.py", line 101, in run
    self._run(pipeline, catalog, run_id)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/runner/sequential_runner.py", line 90, in _run
    run_node(node, catalog, self._is_async, run_id)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/runner/runner.py", line 213, in run_node
    node = _run_node_sequential(node, catalog, run_id)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/runner/runner.py", line 249, in _run_node_sequential
    catalog.save(name, data)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/io/data_catalog.py", line 439, in save
    func(data)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/io/core.py", line 625, in save
    super().save(data)
  File "/home/daniel/git/cotton_counter/.venv/lib/python3.7/site-packages/kedro/io/core.py", line 247, in save
    raise DataSetError(message) from exc
kedro.io.core.DataSetError: Failed while saving data to data set TensorFlowModelDataset(filepath=/home/daniel/git/cotton_counter/data/06_models/fully_trained.hd5, load_args={'compile': False}, protocol=file, save_args={'save_format': h5}, version=Version(load=None, save='2020-09-19T16.20.54.312Z')).
[Errno 2] No such file or directory: '/home/daniel/git/cotton_counter/data/06_models/fully_trained.hd5/2020-09-19T16.20.54.312Z/fully_trained.hd5'
```

Digging deeper, it appears that this issue is caused by `TensorFlowModelDataset` not properly checking to make sure that all intermediate directories are created when saving the model. I was able to fix it by adding two lines to the `_save()` method:

```python
    def _save(self, data: tf.keras.Model) -> None:
        save_path = get_filepath_str(self._get_save_path(), self._protocol)

        # New lines are here.
        save_dir = Path(save_path).parent
        save_dir.mkdir(parents=True, exist_ok=True)
```

I can submit this as a PR also.

## Your Environment
Include as many relevant details about the environment in which you experienced the bug:

* Kedro version used (`pip show kedro` or `kedro -V`): 0.16.5
* Python version used (`python -V`): 3.7.7
* Operating system and version: Ubuntu 20.04



Discussion:

Resolving patch:
diff --git a/kedro/extras/datasets/tensorflow/tensorflow_model_dataset.py b/kedro/extras/datasets/tensorflow/tensorflow_model_dataset.py
--- a/kedro/extras/datasets/tensorflow/tensorflow_model_dataset.py
+++ b/kedro/extras/datasets/tensorflow/tensorflow_model_dataset.py
@@ -31,7 +31,7 @@
 """
 import copy
 import tempfile
-from pathlib import PurePath, PurePosixPath
+from pathlib import Path, PurePath, PurePosixPath
 from typing import Any, Dict
 
 import fsspec
@@ -151,6 +151,10 @@ def _load(self) -> tf.keras.Model:
     def _save(self, data: tf.keras.Model) -> None:
         save_path = get_filepath_str(self._get_save_path(), self._protocol)
 
+        # Make sure all intermediate directories are created.
+        save_dir = Path(save_path).parent
+        save_dir.mkdir(parents=True, exist_ok=True)
+
         with tempfile.TemporaryDirectory(prefix=self._tmp_prefix) as path:
             if self._is_h5:
                 path = str(PurePath(path) / TEMPORARY_H5_FILE)


Test patch:
diff --git a/tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py b/tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py
--- a/tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py
+++ b/tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py
@@ -317,6 +317,30 @@ def test_save_and_load(
         new_predictions = reloaded.predict(dummy_x_test)
         np.testing.assert_allclose(predictions, new_predictions, rtol=1e-6, atol=1e-6)
 
+    def test_hdf5_save_format(
+        self,
+        dummy_tf_base_model,
+        dummy_x_test,
+        filepath,
+        tensorflow_model_dataset,
+        load_version,
+        save_version,
+    ):
+        """Test versioned TensorflowModelDataset can save TF graph models in
+        HDF5 format"""
+        hdf5_dataset = tensorflow_model_dataset(
+            filepath=filepath,
+            save_args={"save_format": "h5"},
+            version=Version(load_version, save_version),
+        )
+
+        predictions = dummy_tf_base_model.predict(dummy_x_test)
+        hdf5_dataset.save(dummy_tf_base_model)
+
+        reloaded = hdf5_dataset.load()
+        new_predictions = reloaded.predict(dummy_x_test)
+        np.testing.assert_allclose(predictions, new_predictions, rtol=1e-6, atol=1e-6)
+
     def test_prevent_overwrite(self, dummy_tf_base_model, versioned_tf_model_dataset):
         """Check the error when attempting to override the data set if the
         corresponding file for a given save version already exists."""