Repository: kedro-org/kedro

Issue:
ShareMemoryDataset does not have `exists()` method
## Description
![image](https://github.com/kedro-org/kedro/assets/18221871/19ecb244-127e-4ffd-82c8-f2854ae83459)

## Context
Unclear, the `SharedMemoryDataset` is implemented by the core team and generating warning when run with `ParallelRunner`, is this false alarm or something we need to address? Cc @merelcht 

## Steps to Reproduce
I was testing something else and see the warning with this command:
`pytest tests/framework/session/test_session_extension_hooks.py::TestNodeHooks::test_on_node_error_hook_parallel_runner`

# Possible Solution:
https://github.com/kedro-org/kedro/blob/0fc8089b637a0679f71e2bddc91f0676fc2914a2/kedro/io/memory_dataset.py#L74C1-L75C40
We have a simple check in `MemoryDataset`, we could implement simliar logic for `ShareMemoryDataset`.

Check whether it makes sense to have an `_exists()` method for `SharedMemoryDataset`. If not: update the logging level to be of `DEBUG`.


Discussion:
The exists() method is not an abstract method and so doesn't have to be implemented. The original `SharedMemoryDataset` was wrapped around `MemoryDataset` directly so I guess that's why the warning wasn't triggered before https://github.com/kedro-org/kedro/pull/3332/files#diff-cc4e916905ba553d257700ff851973772c0d77018d7673a2c4b22f309d8c3e46. IMO we can just ignore this. 
@merelcht Haven't look into this in details, would the alternative be changing that WARNING message to `INFO` instead?
Reported again recently: https://github.com/kedro-org/kedro/issues/4060

Updated the scope to double check where `exists` is called and if it makes sense to implement this method on `SharedMemoryDataset`.

Also related: https://github.com/kedro-org/kedro/issues/4078
Hi, I investigated this for a bit and I believe the issue is that `SharedMemoryDataset` delegates most methods to the underlying `MemoryDataset` via `__getattr__`.
https://github.com/kedro-org/kedro/blob/56e56a783e4b79e96d09c20918a23bb3e11976d1/kedro/io/shared_memory_dataset.py#L33-L37

However, `__getattr__` is only called for attributes that do not exist on the object and since `SharedMemoryDataset` inherits `AbstractDataset._exists` `__getattr__` is never called.

We could implement `_exists` on `SharedMemoryDataset`, we would have to handle the case where `self.shared_memory_dataset` is `None` (return `False` in this case?):

```python
class SharedMemoryDataset(AbstractDataset):
    ...
    def _exists(self) -> bool:
        if not self.shared_memory_dataset:
            return False
        return self.shared_memory_dataset.exists()
```

I'm happy to submit a PR implementing this:)



Patch:
diff --git a/RELEASE.md b/RELEASE.md
index da18b8d172..61560acf87 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -14,6 +14,7 @@
 * Enhanced `OmegaConfigLoader` configuration validation to detect duplicate keys at all parameter levels, ensuring comprehensive nested key checking.
 ## Bug fixes and other changes
 * Fixed bug where using dataset factories breaks with `ThreadRunner`.
+* Fixed a bug where `SharedMemoryDataset.exists` would not call the underlying `MemoryDataset`.
 * Fixed template projects example tests.
 * Made credentials loading consistent between `KedroContext._get_catalog()` and `resolve_patterns` so that both us
 e `_get_config_credentials()`
@@ -30,6 +31,7 @@ e `_get_config_credentials()`
 * [ethanknights](https://github.com/ethanknights)
 * [Manezki](https://github.com/Manezki)
 * [MigQ2](https://github.com/MigQ2)
+* [Felix Scherz](https://github.com/felixscherz)
 
 # Release 0.19.8
 
diff --git a/kedro/io/shared_memory_dataset.py b/kedro/io/shared_memory_dataset.py
index a7e28d0256..139180b578 100644
--- a/kedro/io/shared_memory_dataset.py
+++ b/kedro/io/shared_memory_dataset.py
@@ -57,3 +57,8 @@ def save(self, data: Any) -> None:
     def _describe(self) -> dict[str, Any]:
         """SharedMemoryDataset doesn't have any constructor argument to return."""
         return {}
+
+    def _exists(self) -> bool:
+        if not self.shared_memory_dataset:
+            return False
+        return self.shared_memory_dataset.exists()  # type: ignore[no-any-return]


Test patch:
diff --git a/tests/io/test_shared_memory_dataset.py b/tests/io/test_shared_memory_dataset.py
index d135b3aadd..a4b3526aa4 100644
--- a/tests/io/test_shared_memory_dataset.py
+++ b/tests/io/test_shared_memory_dataset.py
@@ -79,3 +79,9 @@ def test_saving_none(self, shared_memory_dataset):
     def test_str_representation(self, shared_memory_dataset):
         """Test string representation of the dataset"""
         assert "MemoryDataset" in str(shared_memory_dataset)
+
+    def test_exists(self, shared_memory_dataset, input_data):
+        """Check that exists returns the expected values"""
+        assert not shared_memory_dataset.exists()
+        shared_memory_dataset.save(input_data)
+        assert shared_memory_dataset.exists()