Repository: kedro-org/kedro

Issue:
Can we remove `LambdaDataset`?
## Description
I'm wondering if anyone is using the `LambdaDataset` at all.

## Context
While going through the codebase to see what parts need work before Kedro `1.0.0` I came across `LambdaDataset`. Other than linting/typing changes, it hasn't been touched since 2019.



Discussion:
Just my two cents - I never ever see someone use it. I think it well become less useful isf we simplify the creation of a custom dataset (including removing the number of abstract methods like ``_describe``, mutualizing some part of fsspec code in the abstract class...)
We are giving the community a few days to weigh in ahead of deprecating `LambdaDataset`, pending removal in Kedro 0.20.

To note, this section https://docs.kedro.org/en/stable/nodes_and_pipelines/run_a_pipeline.html#output-to-a-file needs to be deleted or completely rewritten.



Patch:
diff --git a/RELEASE.md b/RELEASE.md
index 2599688a4a..c3c78fe0a2 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -103,6 +103,7 @@ Many thanks to the following Kedroids for contributing PRs to this release:
 
 ## Upcoming deprecations for Kedro 0.20.0
 * The utility method `get_pkg_version()` is deprecated and will be removed in Kedro 0.20.0.
+* `LambdaDataset` is deprecated and will be removed in Kedro 0.20.0.
 
 ## Documentation changes
 * Improved documentation for configuring dataset parameters in the data catalog
diff --git a/docs/source/nodes_and_pipelines/run_a_pipeline.md b/docs/source/nodes_and_pipelines/run_a_pipeline.md
index 2bf1a99383..5f17d6e9fc 100644
--- a/docs/source/nodes_and_pipelines/run_a_pipeline.md
+++ b/docs/source/nodes_and_pipelines/run_a_pipeline.md
@@ -238,6 +238,10 @@ Out[11]: {'v': 0.666666666666667}
 
 We can also use IO to save outputs to a file. In this example, we define a custom `LambdaDataset` that would serialise the output to a file locally:
 
+```{warning}
+`LambdaDataset` has been deprecated and will be removed in Kedro `0.20.0`.
+```
+
 <details>
 <summary><b>Click to expand</b></summary>
 
diff --git a/kedro/io/lambda_dataset.py b/kedro/io/lambda_dataset.py
index d120f74ed2..4e1739114d 100644
--- a/kedro/io/lambda_dataset.py
+++ b/kedro/io/lambda_dataset.py
@@ -5,8 +5,10 @@
 
 from __future__ import annotations
 
+import warnings
 from typing import Any, Callable
 
+from kedro import KedroDeprecationWarning
 from kedro.io.core import AbstractDataset, DatasetError
 
 
@@ -100,6 +102,12 @@ def __init__(
             DatasetError: If a method is specified, but is not a Callable.
 
         """
+
+        warnings.warn(
+            "`LambdaDataset` has been deprecated and will be removed in Kedro 0.20.0.",
+            KedroDeprecationWarning,
+        )
+
         for name, value in [
             ("load", load),
             ("save", save),


Test patch:
diff --git a/tests/io/test_lambda_dataset.py b/tests/io/test_lambda_dataset.py
index eac9709d04..ca064ade36 100644
--- a/tests/io/test_lambda_dataset.py
+++ b/tests/io/test_lambda_dataset.py
@@ -1,5 +1,6 @@
 import pytest
 
+from kedro import KedroDeprecationWarning
 from kedro.io import DatasetError, LambdaDataset
 
 
@@ -81,6 +82,14 @@ def test_ephemeral_attribute(mocked_dataset):
     assert mocked_dataset._EPHEMERAL is False
 
 
+def test_lambda_dataset_deprecation():
+    with pytest.warns(
+        KedroDeprecationWarning,
+        match=r"\`LambdaDataset\` has been deprecated",
+    ):
+        _ = LambdaDataset(None, None)
+
+
 class TestLambdaDatasetLoad:
     def test_load_invocation(self, mocker):
         """Test the basic `load` method invocation"""