Repository: deepset-ai/haystack
Pull request: feat: Load documents from remote - helper function
First step of the Community Sprint issue I'm working on with @recrudesce  #2372
- introduces a helper function in utils to 'load dataset from remote'
- Checks that the dataset exists
- Checks that the dataset has the required fields to _be_ a Haystack document

Can someone guide us as to how we should handle esceptions? Currently I do a very ugly 'print'

I think we should have this helper function any way as an easy first step. Any custom node we decide to create that would pull documents straight in from Hugging Face would be able to make use of this function too cc: @mayankjobanputra 

### Checklist
- [x] I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- [x] I have updated the related issue with new insights and changes
- [ ] I added tests that demonstrate the correct behavior of the change
- [x] I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- [x] I documented my code
- [x] I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Add a helper function to get datasets from HF and write them to a DocumentStore
As suggested by @vblagoje on the Haystack slack, it would be nice to have a helper function, similar to `open_search_index_to_documentstore` or `convert_files_to_docs` that would allow users to provide a HF dataset name and that would write them in Document format to a DocumentStore

Hints from issue discussion:
Hey @ZanSara, I'm a research engineer working on language modelling and wanting to contribute to open source. I was wondering if this is still open and if I could try to implement a solution?
Hey @jackapbutler - it is still open actually yes :) - me and @mayankjobanputra were working on an implementation idea. Our current idea has gone beyond a helper function and potentially providing a node. Would you like to create a proposal for this? 
I am working on this issue during our first Community Sprint #4489 

There are 2 parts to this feature:
1. Being able to push an array Haystack Documents to a HuggingFace repository
2. Being able to pull.

I propose to focus first on step 2 and create a helper function that e.g:
`pull_documents_from_remote(dataset_name)` --> returns list of documents that we can then pass to `write_documents`
OR
We can change the `write documents` `documents` argument also accept a repo name from HF.

Any ideas here? 
Hello!
Just my two cents:

> `pull_documents_from_remote(dataset_name)` --> returns list of documents that we can then pass to `write_documents`

👍 IMO, this idea allows for more flexibility and lets `write_documents` focus on its scope only.

Commit message: fixed black

Patch to review:
diff --git a/haystack/utils/import_utils.py b/haystack/utils/import_utils.py
index 5a1385a2e9..de92eaa3bb 100644
--- a/haystack/utils/import_utils.py
+++ b/haystack/utils/import_utils.py
@@ -1,5 +1,3 @@
-from typing import Optional, Dict, Union, Tuple
-
 import io
 import gzip
 import tarfile
@@ -8,8 +6,11 @@
 import importlib
 import importlib.util
 from pathlib import Path
+from typing import Optional, Dict, Union, Tuple, List
 
 import requests
+from haystack.errors import HaystackError
+from haystack.schema import Document
 
 
 logger = logging.getLogger(__name__)
@@ -62,6 +63,30 @@ def _optional_component_not_installed(component: str, dep_group: str, source_err
     ) from source_error
 
 
+def load_documents_from_datasets(dataset_name: str, split: Optional[str] = "train") -> List[Document]:
+    """
+    Load a list of Haystack Documents from a remote Hugging Face dataset.
+
+    :param dataset_name: A Hugging Face dataset containing Haystack Documents
+    :param split: The split of the Hugging Face dataset to load from. By default, this is set to "train".
+    :return: a List of Haystack Documents
+    """
+    try:
+        from datasets import load_dataset, load_dataset_builder
+    except ImportError:
+        raise ImportError(
+            "Failed to import `datasets`, Run 'pip install datasets>=2.6.0' "
+            "to install the datasets library to use this function."
+        )
+
+    dataset = load_dataset_builder(dataset_name)
+    if "content" not in dataset.info.features.keys():
+        raise HaystackError("Dataset does not contain a content field which is required by Haystack Documents")
+
+    remote_dataset = load_dataset(dataset_name, split=split)
+    return remote_dataset
+
+
 def fetch_archive_from_http(
     url: str,
     output_dir: str,


Code review comments:
diff_hunk: @@ -62,6 +63,30 @@ def _optional_component_not_installed(component: str, dep_group: str, source_err
     ) from source_error
 
 
+def load_documents_from_datasets(dataset_name: str, split: Optional[str] = "train") -> List[Document]:
+    """
+    Load a list of Haystack Documents from a remote Hugging Face dataset.
+
+    :param dataset_name: A Hugging Face dataset containing Haystack Documents
+    :param split: The split of the Hugging Face dataset to load from. By default, this is set to "train".
+    :return: a List of Haystack Documents
+    """
+    try:
+        from datasets import load_dataset, load_dataset_builder
+    except ImportError:
+        raise ImportError(
+            "Failed to import `datasets`, Run 'pip install datasets>=2.6.0' "
+            "to install the datasets library to use this function."
+        )
+
+    dataset = load_dataset_builder(dataset_name)
+    if "content" not in dataset.info.features.keys():
+        raise HaystackError("Dataset does not contain a content field which is required by Haystack Documents")
+
+    remote_dataset = load_dataset(dataset_name, split=split)
path: haystack/utils/import_utils.py
text: @user1:
@author this returns a `Dataset` instance and not a list of Haystack `Documents`, is that what we want?

@author:
@user1 good catch. This was not my intention. I just pushed what I _think_ could be a way to handle this. Let me know what you think

@user1:
I just had a small nit-pick which I fixed. Good to go for me now.