Repository: deepset-ai/haystack
Pull request: Adding yaml functionality to standard pipelines (save/load...)
**Proposed changes**:
- adds relevant methods to BaseStandardPipeline to support loading from and saving to yaml files
- closes #1681

**Status (please check what you already did)**:
- [x] First draft (up for discussions & feedback)
- [ ] Final code
- [ ] Added tests
- [ ] Updated documentation

This would make further work necessary for SearchSummarizationPipeline and MostSimilarDocumentsPipeline as they additional attributes to self.pipeline.

The attribute in SearchSummarizationPipeline only exists because the summarizer has an untypical return format. We could either change the return value or override the load and save method. Alternatively, we could also just override the load method and add a parameter to the load method specifying the return type.

For MostSimilarDocumentsPipeline, the problem is that it isn't actually using a pipeline. We could either add a "wrapper" pipeline in the class with no additional functionality or override save and load methods.

Problem statement:
Easier way to generate a YAML file from a standard pipeline
Since the standard pipelines don't inherit from `Pipeline` but rather from `BaseStandardPipeline`, they don't have a `save_to_yaml()' method. There have been users in our community who want to create a yaml for these standard pipelines. We should make it easier for them to generate the yaml to use in a REST API application.

Hints from issue discussion:
Let's add these methods to `BaseStandardPipeline`:
- `save_to_yaml()`
- `load_from_yaml()`
- `get_nodes_by_class()`
- `get_document_store()`

@MichelBartels They can simply forward the call to the respective methods of the `Pipeline`class. Similar to the already existing functions here: https://github.com/deepset-ai/haystack/blob/5ac89332a29f2e75c7859f481976ba2137c28489/haystack/pipelines/standard_pipelines.py#L45

Commit message: Add latest docstring and tutorial changes

Patch to review:
diff --git a/docs/_src/api/api/pipelines.md b/docs/_src/api/api/pipelines.md
index 3faa60f55f..680dc1b1c3 100644
--- a/docs/_src/api/api/pipelines.md
+++ b/docs/_src/api/api/pipelines.md
@@ -481,6 +481,102 @@ Create a Graphviz visualization of the pipeline.
 
 - `path`: the path to save the image.
 
+<a name="standard_pipelines.BaseStandardPipeline.save_to_yaml"></a>
+#### save\_to\_yaml
+
+```python
+ | save_to_yaml(path: Path, return_defaults: bool = False)
+```
+
+Save a YAML configuration for the Pipeline that can be used with `Pipeline.load_from_yaml()`.
+
+**Arguments**:
+
+- `path`: path of the output YAML file.
+- `return_defaults`: whether to output parameters that have the default values.
+
+<a name="standard_pipelines.BaseStandardPipeline.load_from_yaml"></a>
+#### load\_from\_yaml
+
+```python
+ | @classmethod
+ | load_from_yaml(cls, path: Path, pipeline_name: Optional[str] = None, overwrite_with_env_variables: bool = True)
+```
+
+Load Pipeline from a YAML file defining the individual components and how they're tied together to form
+a Pipeline. A single YAML can declare multiple Pipelines, in which case an explicit `pipeline_name` must
+be passed.
+
+Here's a sample configuration:
+
+    ```yaml
+    |   version: '0.8'
+    |
+    |    components:    # define all the building-blocks for Pipeline
+    |    - name: MyReader       # custom-name for the component; helpful for visualization & debugging
+    |      type: FARMReader    # Haystack Class name for the component
+    |      params:
+    |        no_ans_boost: -10
+    |        model_name_or_path: deepset/roberta-base-squad2
+    |    - name: MyESRetriever
+    |      type: ElasticsearchRetriever
+    |      params:
+    |        document_store: MyDocumentStore    # params can reference other components defined in the YAML
+    |        custom_query: null
+    |    - name: MyDocumentStore
+    |      type: ElasticsearchDocumentStore
+    |      params:
+    |        index: haystack_test
+    |
+    |    pipelines:    # multiple Pipelines can be defined using the components from above
+    |    - name: my_query_pipeline    # a simple extractive-qa Pipeline
+    |      nodes:
+    |      - name: MyESRetriever
+    |        inputs: [Query]
+    |      - name: MyReader
+    |        inputs: [MyESRetriever]
+    ```
+
+**Arguments**:
+
+- `path`: path of the YAML file.
+- `pipeline_name`: if the YAML contains multiple pipelines, the pipeline_name to load must be set.
+- `overwrite_with_env_variables`: Overwrite the YAML configuration with environment variables. For example,
+                                     to change index name param for an ElasticsearchDocumentStore, an env
+                                     variable 'MYDOCSTORE_PARAMS_INDEX=documents-2021' can be set. Note that an
+                                     `_` sign must be used to specify nested hierarchical properties.
+
+<a name="standard_pipelines.BaseStandardPipeline.get_nodes_by_class"></a>
+#### get\_nodes\_by\_class
+
+```python
+ | get_nodes_by_class(class_type) -> List[Any]
+```
+
+Gets all nodes in the pipeline that are an instance of a certain class (incl. subclasses).
+This is for example helpful if you loaded a pipeline and then want to interact directly with the document store.
+Example:
+| from haystack.document_stores.base import BaseDocumentStore
+| INDEXING_PIPELINE = Pipeline.load_from_yaml(Path(PIPELINE_YAML_PATH), pipeline_name=INDEXING_PIPELINE_NAME)
+| res = INDEXING_PIPELINE.get_nodes_by_class(class_type=BaseDocumentStore)
+
+**Returns**:
+
+List of components that are an instance the requested class
+
+<a name="standard_pipelines.BaseStandardPipeline.get_document_store"></a>
+#### get\_document\_store
+
+```python
+ | get_document_store() -> Optional[BaseDocumentStore]
+```
+
+Return the document store object used in the current pipeline.
+
+**Returns**:
+
+Instance of DocumentStore or None
+
 <a name="standard_pipelines.ExtractiveQAPipeline"></a>
 ## ExtractiveQAPipeline Objects
 
diff --git a/haystack/pipelines/standard_pipelines.py b/haystack/pipelines/standard_pipelines.py
index cb1bc7a8a6..67372ad2ad 100644
--- a/haystack/pipelines/standard_pipelines.py
+++ b/haystack/pipelines/standard_pipelines.py
@@ -2,7 +2,7 @@
 from abc import ABC
 from copy import deepcopy
 from pathlib import Path
-from typing import List, Optional, Dict
+from typing import List, Optional, Dict, Any
 from functools import wraps
 
 from haystack.schema import Document
@@ -69,6 +69,84 @@ def draw(self, path: Path = Path("pipeline.png")):
         :param path: the path to save the image.
         """
         self.pipeline.draw(path)
+    
+    def save_to_yaml(self, path: Path, return_defaults: bool = False):
+        """
+        Save a YAML configuration for the Pipeline that can be used with `Pipeline.load_from_yaml()`.
+
+        :param path: path of the output YAML file.
+        :param return_defaults: whether to output parameters that have the default values.
+        """
+        return self.pipeline.save_to_yaml(path, return_defaults)
+    
+    @classmethod
+    def load_from_yaml(cls, path: Path, pipeline_name: Optional[str] = None, overwrite_with_env_variables: bool = True):
+        """
+        Load Pipeline from a YAML file defining the individual components and how they're tied together to form
+        a Pipeline. A single YAML can declare multiple Pipelines, in which case an explicit `pipeline_name` must
+        be passed.
+
+        Here's a sample configuration:
+
+            ```yaml
+            |   version: '0.8'
+            |
+            |    components:    # define all the building-blocks for Pipeline
+            |    - name: MyReader       # custom-name for the component; helpful for visualization & debugging
+            |      type: FARMReader    # Haystack Class name for the component
+            |      params:
+            |        no_ans_boost: -10
+            |        model_name_or_path: deepset/roberta-base-squad2
+            |    - name: MyESRetriever
+            |      type: ElasticsearchRetriever
+            |      params:
+            |        document_store: MyDocumentStore    # params can reference other components defined in the YAML
+            |        custom_query: null
+            |    - name: MyDocumentStore
+            |      type: ElasticsearchDocumentStore
+            |      params:
+            |        index: haystack_test
+            |
+            |    pipelines:    # multiple Pipelines can be defined using the components from above
+            |    - name: my_query_pipeline    # a simple extractive-qa Pipeline
+            |      nodes:
+            |      - name: MyESRetriever
+            |        inputs: [Query]
+            |      - name: MyReader
+            |        inputs: [MyESRetriever]
+            ```
+
+        :param path: path of the YAML file.
+        :param pipeline_name: if the YAML contains multiple pipelines, the pipeline_name to load must be set.
+        :param overwrite_with_env_variables: Overwrite the YAML configuration with environment variables. For example,
+                                             to change index name param for an ElasticsearchDocumentStore, an env
+                                             variable 'MYDOCSTORE_PARAMS_INDEX=documents-2021' can be set. Note that an
+                                             `_` sign must be used to specify nested hierarchical properties.
+        """
+        self = cls.__new__(cls)
+        self.pipeline = Pipeline.load_from_yaml(path, pipeline_name, overwrite_with_env_variables)
+        return self
+    
+    def get_nodes_by_class(self, class_type) -> List[Any]:
+        """
+        Gets all nodes in the pipeline that are an instance of a certain class (incl. subclasses).
+        This is for example helpful if you loaded a pipeline and then want to interact directly with the document store.
+        Example:
+        | from haystack.document_stores.base import BaseDocumentStore
+        | INDEXING_PIPELINE = Pipeline.load_from_yaml(Path(PIPELINE_YAML_PATH), pipeline_name=INDEXING_PIPELINE_NAME)
+        | res = INDEXING_PIPELINE.get_nodes_by_class(class_type=BaseDocumentStore)
+
+        :return: List of components that are an instance the requested class
+        """
+        return self.pipeline.get_nodes_by_class(class_type)
+    
+    def get_document_store(self) -> Optional[BaseDocumentStore]:
+        """
+        Return the document store object used in the current pipeline.
+
+        :return: Instance of DocumentStore or None
+        """
+        return self.pipeline.get_document_store()
 
 
 class ExtractiveQAPipeline(BaseStandardPipeline):


Code review comments:
diff_hunk: @@ -69,6 +69,84 @@ def draw(self, path: Path = Path("pipeline.png")):
         :param path: the path to save the image.
         """
         self.pipeline.draw(path)
+    
+    def save_to_yaml(self, path: Path, return_defaults: bool = False):
+        """
+        Save a YAML configuration for the Pipeline that can be used with `Pipeline.load_from_yaml()`.
+
+        :param path: path of the output YAML file.
+        :param return_defaults: whether to output parameters that have the default values.
+        """
+        return self.pipeline.save_to_yaml(path, return_defaults)
+    
+    @classmethod
+    def load_from_yaml(cls, path: Path, pipeline_name: Optional[str] = None, overwrite_with_env_variables: bool = True):
+        """
+        Load Pipeline from a YAML file defining the individual components and how they're tied together to form
+        a Pipeline. A single YAML can declare multiple Pipelines, in which case an explicit `pipeline_name` must
+        be passed.
+
+        Here's a sample configuration:
+
+            ```yaml
+            |   version: '0.8'
+            |
+            |    components:    # define all the building-blocks for Pipeline
+            |    - name: MyReader       # custom-name for the component; helpful for visualization & debugging
+            |      type: FARMReader    # Haystack Class name for the component
+            |      params:
+            |        no_ans_boost: -10
+            |        model_name_or_path: deepset/roberta-base-squad2
+            |    - name: MyESRetriever
+            |      type: ElasticsearchRetriever
+            |      params:
+            |        document_store: MyDocumentStore    # params can reference other components defined in the YAML
+            |        custom_query: null
+            |    - name: MyDocumentStore
+            |      type: ElasticsearchDocumentStore
+            |      params:
+            |        index: haystack_test
+            |
+            |    pipelines:    # multiple Pipelines can be defined using the components from above
+            |    - name: my_query_pipeline    # a simple extractive-qa Pipeline
+            |      nodes:
+            |      - name: MyESRetriever
+            |        inputs: [Query]
+            |      - name: MyReader
+            |        inputs: [MyESRetriever]
+            ```
+
+        :param path: path of the YAML file.
+        :param pipeline_name: if the YAML contains multiple pipelines, the pipeline_name to load must be set.
+        :param overwrite_with_env_variables: Overwrite the YAML configuration with environment variables. For example,
+                                             to change index name param for an ElasticsearchDocumentStore, an env
+                                             variable 'MYDOCSTORE_PARAMS_INDEX=documents-2021' can be set. Note that an
+                                             `_` sign must be used to specify nested hierarchical properties.
+        """
+        self = cls.__new__(cls)
path: haystack/pipelines/standard_pipelines.py
text: @user1:
We should add a comment here explaining why this is necessary. In addition, I wonder whether we can find a better variable name than `self` because the method is a class method so I at first expected `self` to be a class but we are creating a new instance of that class here. Maybe something like `standard_pipeline_object`?

@author:
I also changed that.

diff_hunk: @@ -69,6 +69,84 @@ def draw(self, path: Path = Path("pipeline.png")):
         :param path: the path to save the image.
         """
         self.pipeline.draw(path)
+    
+    def save_to_yaml(self, path: Path, return_defaults: bool = False):
+        """
+        Save a YAML configuration for the Pipeline that can be used with `Pipeline.load_from_yaml()`.
+
+        :param path: path of the output YAML file.
+        :param return_defaults: whether to output parameters that have the default values.
+        """
+        return self.pipeline.save_to_yaml(path, return_defaults)
+    
+    @classmethod
+    def load_from_yaml(cls, path: Path, pipeline_name: Optional[str] = None, overwrite_with_env_variables: bool = True):
+        """
+        Load Pipeline from a YAML file defining the individual components and how they're tied together to form
+        a Pipeline. A single YAML can declare multiple Pipelines, in which case an explicit `pipeline_name` must
+        be passed.
+
+        Here's a sample configuration:
+
+            ```yaml
+            |   version: '0.8'
+            |
+            |    components:    # define all the building-blocks for Pipeline
+            |    - name: MyReader       # custom-name for the component; helpful for visualization & debugging
+            |      type: FARMReader    # Haystack Class name for the component
+            |      params:
+            |        no_ans_boost: -10
+            |        model_name_or_path: deepset/roberta-base-squad2
+            |    - name: MyESRetriever
+            |      type: ElasticsearchRetriever
+            |      params:
+            |        document_store: MyDocumentStore    # params can reference other components defined in the YAML
+            |        custom_query: null
+            |    - name: MyDocumentStore
+            |      type: ElasticsearchDocumentStore
+            |      params:
+            |        index: haystack_test
+            |
+            |    pipelines:    # multiple Pipelines can be defined using the components from above
+            |    - name: my_query_pipeline    # a simple extractive-qa Pipeline
+            |      nodes:
+            |      - name: MyESRetriever
+            |        inputs: [Query]
+            |      - name: MyReader
+            |        inputs: [MyESRetriever]
+            ```
+
+        :param path: path of the YAML file.
+        :param pipeline_name: if the YAML contains multiple pipelines, the pipeline_name to load must be set.
+        :param overwrite_with_env_variables: Overwrite the YAML configuration with environment variables. For example,
+                                             to change index name param for an ElasticsearchDocumentStore, an env
+                                             variable 'MYDOCSTORE_PARAMS_INDEX=documents-2021' can be set. Note that an
+                                             `_` sign must be used to specify nested hierarchical properties.
+        """
+        self = cls.__new__(cls)
+        self.pipeline = Pipeline.load_from_yaml(path, pipeline_name, overwrite_with_env_variables)
+        return self
+    
+    def get_nodes_by_class(self, class_type) -> List[Any]:
+        """
+        Gets all nodes in the pipeline that are an instance of a certain class (incl. subclasses).
+        This is for example helpful if you loaded a pipeline and then want to interact directly with the document store.
+        Example:
+        | from haystack.document_stores.base import BaseDocumentStore
+        | INDEXING_PIPELINE = Pipeline.load_from_yaml(Path(PIPELINE_YAML_PATH), pipeline_name=INDEXING_PIPELINE_NAME)
+        | res = INDEXING_PIPELINE.get_nodes_by_class(class_type=BaseDocumentStore)
+
+        :return: List of components that are an instance the requested class
path: haystack/pipelines/standard_pipelines.py
text: @user1:
small typo: "instance the" -> "instance of the"

@author:
I made the change.