Repository: argilla-io/distilabel

Issue:
How to load local hf dataset？
Hi, I have run the example provided on this [page](https://github.com/argilla-io/distilabel?tab=readme-ov-file#example) and it run well. Then I tried to load a local HF dataset and found that the class LoadDataFromDisk. 

I saved the dataset using the `save_to_disk` method like `dataset.save_to_disk(dataset_dict_path='./save_data') ` and following is the structure of the saved dataset directory.

save_data/
├── dataset_dict.json
└── test
    ├── data-00000-of-00001.arrow
    ├── dataset_info.json
    └── state.json

I only changed the "repo_id" to "dataset_path" (a local LLM is used). When I run the following code, a Value error encountered 
```
ValueError: Step 'load_data_from_disk_0' is missing required runtime parameter 'repo_id'. Please, provide a value for it when calling `Pipeline.run` method:

    pipeline.run(parameters={"load_data_from_disk_0": {"repo_id": ...}})
```

```
from distilabel.llms import vLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromDisk
from distilabel.steps.tasks import TextGeneration

with Pipeline(
    name="simple-text-generation-pipeline",
    description="A simple text generation pipeline",
) as pipeline:
    load_dataset = LoadDataFromDisk(output_mappings={"prompt": "instruction"})

    generate_with_openai = TextGeneration(llm=vLLM(model="/home/public_data/qwen/Qwen2-7B-Instruct/"))

    load_dataset >> generate_with_openai

if __name__ == "__main__":
    distiset = pipeline.run(
        parameters={
            load_dataset.name: {
                "dataset_path": "/home/qiey/DataGenration/save_data",
                "split": "test",
            },
            generate_with_openai.name: {
                "llm": {
                    "generation_kwargs": {
                        "temperature": 0.7,
                        "max_new_tokens": 512,
                    }
                }
            },
        },
    )

```
could anyone give me some advice, any suggestion is appreciated.

Discussion:
@gabrielmbmb: Hi @yuqie, thanks for reporting the issue! It looks like the `repo_id` attribute is mandatory in the `LoadDataFromDisk` because it's inherited from `LoadDataFromHub` (it shouldn't be this way). In the meantime we fix the issue, you can by pass this error passing `LoadDataFromDisk(repo_id="random", ...)`. Related to #898 cc @plaguss @yuqie: Hi @gabrielmbmb!,  thanks for your help. I have tried to pass "random" to repo_id, it works when I tried load the dataset alone.
```
from distilabel.steps import LoadDataFromDisk
load_dataset = LoadDataFromDisk(repo_id="random",dataset_path="/home/qiey/DataGenration/save_data",output_mappings={"prompt": "instruction"})
```
If I used it with the pipeline, a ValueError which required the parameter 'config' is encountered.
```
ValueError: Step 'load_data_from_disk_0' is missing required runtime parameter 'config'. Please, provide a value for it when calling `Pipeline.run` method:

    pipeline.run(parameters={"load_data_from_disk_0": {"config": ...}})

```
Here is the whole code.
```
from distilabel.llms import vLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromDisk
from distilabel.steps.tasks import TextGeneration

with Pipeline(
    name="simple-text-generation-pipeline",
    description="A simple text generation pipeline",
) as pipeline:
    load_dataset = LoadDataFromDisk(repo_id="random",dataset_path="/home/qiey/DataGenration/save_data",split="test",output_mappings={"prompt": "instruction"})

    generate_with_openai = TextGeneration(llm=vLLM(model="/home/public_data/qwen/Qwen2-7B-Instruct/"))

    load_dataset >> generate_with_openai

if __name__ == "__main__":
    distiset = pipeline.run(
        parameters={
            generate_with_openai.name: {
                "llm": {
                    "generation_kwargs": {
                        "temperature": 0.7,
                        "max_new_tokens": 512,
                    }
                }
            },
        },
    )
```
I tried to add the parameter config="default", the following is happened:
```
ValueError: Dataset not loaded yet, you must call `load` method first.
```
btw the distibabel version is 1.2.4, and I tried to update to 1.3.1. I run with the thread error no matter local data or hf dataset.
```
RuntimeError: Failed to load all the steps. Could not run pipeline.
Exception in thread Thread-1 (_monitor):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
```
@plaguss: Hi @yuqie, we are working to fix this bug in PR https://github.com/argilla-io/distilabel/pull/910.
Once it's merged, you should be able to use the class as expected. The `load` method will be done automatically if necessary, the config will be optional (it will default to `default`, which is the standard name, but if that's not possible, an error will be raised informing).

Related to the last `RuntimeError`, let's see after trying with the new version, and if that persists, maybe you can try running with the `dry_run` and sharing the full error trace

Update:
The PR is already merged, let us know if you update the code and run it with the new version@plaguss: Closing this, reopen it if you find the error persists, or open a new one if some other error appears 😄 

Solution patch:
diff --git a/src/distilabel/steps/generators/huggingface.py b/src/distilabel/steps/generators/huggingface.py
index 243277fd4..4789dfe10 100644
--- a/src/distilabel/steps/generators/huggingface.py
+++ b/src/distilabel/steps/generators/huggingface.py
@@ -460,16 +460,16 @@ class LoadDataFromDisk(LoadDataFromHub):
     Attributes:
         dataset_path: The path to the dataset or distiset.
         split: The split of the dataset to load (typically will be `train`, `test` or `validation`).
-        config: The configuration of the dataset to load. This is optional and only needed
-            if the dataset has multiple configurations.
+        config: The configuration of the dataset to load. Defaults to `default`, if there are
+            multiple configurations in the dataset this must be suplied or an error is raised.
 
     Runtime parameters:
         - `batch_size`: The batch size to use when processing the data.
         - `dataset_path`: The path to the dataset or distiset.
         - `is_distiset`: Whether the dataset to load is a `Distiset` or not. Defaults to False.
         - `split`: The split of the dataset to load. Defaults to 'train'.
-        - `config`: The configuration of the dataset to load. This is optional and only
-            needed if the dataset has multiple configurations.
+        - `config`: The configuration of the dataset to load. Defaults to `default`, if there are
+            multiple configurations in the dataset this must be suplied or an error is raised.
         - `num_examples`: The number of examples to load from the dataset.
             By default will load all examples.
         - `storage_options`: Key/value pairs to be passed on to the file-system backend, if any.
@@ -539,10 +539,12 @@ class LoadDataFromDisk(LoadDataFromHub):
         default=None,
         description="Path to the dataset or distiset.",
     )
-    config: RuntimeParameter[str] = Field(
-        default=None,
-        description="The configuration of the dataset to load. This is optional and only"
-        " needed if the dataset has multiple configurations.",
+    config: Optional[RuntimeParameter[str]] = Field(
+        default="default",
+        description=(
+            "The configuration of the dataset to load. Will default to 'default'",
+            " which corresponds to a distiset with a single configuration.",
+        ),
     )
     is_distiset: Optional[RuntimeParameter[bool]] = Field(
         default=False,
@@ -557,6 +559,7 @@ class LoadDataFromDisk(LoadDataFromHub):
         default=None,
         description="The split of the dataset to load. By default will load the whole Dataset/Distiset.",
     )
+    repo_id: ExcludedField[Union[str, None]] = None
 
     def load(self) -> None:
         """Load the dataset from the file/s in disk."""
@@ -567,8 +570,15 @@ class LoadDataFromDisk(LoadDataFromHub):
                 keep_in_memory=self.keep_in_memory,
                 storage_options=self.storage_options,
             )
-            if self.config:
-                ds = ds[self.config]
+            if self.config not in ds.keys():
+                # TODO: Add FAQ for this case, pointing to the Distiset documentation on configurations.
+                raise ValueError(
+                    f"Configuration '{self.config}' not found in the Distiset, available ones"
+                    f" are: {list(ds.keys())}. Please try changing the `config` parameter to one "
+                    "of the available configurations.\n\n"
+                    f"For more information on Distiset configurations, please visit https://distilabel.argilla.io/dev/sections/how_to_guides/advanced/distiset/#using-the-distiset-dataset-object"
+                )
+            ds = ds[self.config]
 
         else:
             ds = load_from_disk(
@@ -596,9 +606,7 @@ class LoadDataFromDisk(LoadDataFromHub):
             The columns that will be generated by this step.
         """
         # We assume there are Dataset/IterableDataset, not it's ...Dict counterparts
-        if self._dataset is Ellipsis:
-            raise ValueError(
-                "Dataset not loaded yet, you must call `load` method first."
-            )
+        if self._dataset is None:
+            self.load()
 
         return self._dataset.column_names


Test patch:
diff --git a/tests/unit/steps/generators/test_huggingface.py b/tests/unit/steps/generators/test_huggingface.py
index 281d5187e..23b3b8007 100644
--- a/tests/unit/steps/generators/test_huggingface.py
+++ b/tests/unit/steps/generators/test_huggingface.py
@@ -168,6 +168,32 @@ class TestLoadDataFromDisk:
             assert isinstance(generator_step_output[1], bool)
             assert len(generator_step_output[0]) == 3
 
+    @pytest.mark.parametrize("config_name", ["default", "missnamed_config"])
+    def test_load_distiset_from_disk_default(self, config_name: str) -> None:
+        distiset = Distiset(
+            {
+                "default": Dataset.from_dict({"a": [1, 2, 3]}),
+            }
+        )
+        with tempfile.TemporaryDirectory() as tmpdir:
+            dataset_path = str(Path(tmpdir) / "dataset_path")
+            distiset.save_to_disk(dataset_path)
+
+            loader = LoadDataFromDisk(
+                dataset_path=dataset_path,
+                is_distiset=True,
+                config=config_name,
+            )
+            if config_name != "default":
+                with pytest.raises(ValueError):
+                    loader.load()
+            else:
+                loader.load()
+                generator_step_output = next(loader.process())
+                assert isinstance(generator_step_output, tuple)
+                assert isinstance(generator_step_output[1], bool)
+                assert len(generator_step_output[0]) == 3
+
     def test_load_distiset_from_disk(self) -> None:
         distiset = Distiset(
             {