Repository: deepset-ai/haystack

Issue:
Validation of Ray pipelines
**Problem**
Ray pipelines do not conform fully to the JSON schema we defined for regular Pipelines. Therefore they will often fail validation, even though they are completely valid.

**Solution**
Define a new schema for Ray pipelines and validate them against it. We could heavily leverage the existing JSON schema generation code to do so.

**Alternatives**
Relaxing the schema to allow bot regular and Ray pipelines could lead to a lot of confusion about which attributes are valid where. Also, given that the pipeline type was an attribute value, selecting different structures based on it would have been quite convoluted in the schema. I believe separate schemas are much more clear.



Discussion:
@ZanSara could you please briefly explain why ray pipelines need a special treatment when it comes to the validation? What's the difference in the schemas? ( @oryx1729 is also interested) 🙂 
I decided to separate the validation of Ray pipelines when I noticed that they have an extra parameter in the pipelines attributes: `replicas` https://github.com/deepset-ai/haystack/blob/6fb58d09a9ac0f6895bbbb9e617d459b5f27f155/test/samples/pipeline/test_ray_pipeline.yaml#L38-L44

The fact is that I am not sure how many of these additional params there are, whether they are mandatory, etc. So I thought taking this specific topic out of the main PR was wiser. 

I think all answers are in the code of `RayPipeline` so I don't expect to have trouble with it, but I'll surely reach out to @oryx1729 in case of doubts.

Resolving patch:
diff --git a/haystack/nodes/_json_schema.py b/haystack/nodes/_json_schema.py
--- a/haystack/nodes/_json_schema.py
+++ b/haystack/nodes/_json_schema.py
@@ -249,6 +249,12 @@ def get_json_schema(filename: str, version: str, modules: List[str] = ["haystack
                 "type": "string",
                 "const": version,
             },
+            "extras": {
+                "title": "Additional properties group",
+                "description": "To be specified only if contains special pipelines (for example, if this is a Ray pipeline)",
+                "type": "string",
+                "enum": ["ray"],
+            },
             "components": {
                 "title": "Components",
                 "description": "Component nodes and their configurations, to later be used in the pipelines section. Define here all the building blocks for the pipelines.",
@@ -283,6 +289,11 @@ def get_json_schema(filename: str, version: str, modules: List[str] = ["haystack
                                         "type": "array",
                                         "items": {"type": "string"},
                                     },
+                                    "replicas": {
+                                        "title": "replicas",
+                                        "description": "How many replicas Ray should create for this node (only for Ray pipelines)",
+                                        "type": "integer",
+                                    },
                                 },
                                 "required": ["name", "inputs"],
                                 "additionalProperties": False,
@@ -298,6 +309,18 @@ def get_json_schema(filename: str, version: str, modules: List[str] = ["haystack
         },
         "required": ["version", "components", "pipelines"],
         "additionalProperties": False,
+        "oneOf": [
+            {
+                "not": {"required": ["extras"]},
+                "properties": {
+                    "pipelines": {
+                        "title": "Pipelines",
+                        "items": {"properties": {"nodes": {"items": {"not": {"required": ["replicas"]}}}}},
+                    }
+                },
+            },
+            {"properties": {"extras": {"enum": ["ray"]}}, "required": ["extras"]},
+        ],
         "definitions": schema_definitions,
     }
     return pipeline_schema
diff --git a/haystack/pipelines/ray.py b/haystack/pipelines/ray.py
--- a/haystack/pipelines/ray.py
+++ b/haystack/pipelines/ray.py
@@ -12,7 +12,12 @@
     ray = None  # type: ignore
     serve = None  # type: ignore
 
-from haystack.pipelines.config import get_component_definitions, get_pipeline_definition, read_pipeline_config_from_yaml
+from haystack.pipelines.config import (
+    get_component_definitions,
+    get_pipeline_definition,
+    read_pipeline_config_from_yaml,
+    validate_config,
+)
 from haystack.schema import MultiLabel, Document
 from haystack.nodes.base import BaseComponent, RootNode
 from haystack.pipelines.base import Pipeline
@@ -72,6 +77,8 @@ def load_from_config(
         address: Optional[str] = None,
         ray_args: Optional[Dict[str, Any]] = None,
     ):
+        validate_config(pipeline_config, strict_version_check=strict_version_check)
+
         pipeline_definition = get_pipeline_definition(pipeline_config=pipeline_config, pipeline_name=pipeline_name)
         component_definitions = get_component_definitions(
             pipeline_config=pipeline_config, overwrite_with_env_variables=overwrite_with_env_variables
@@ -165,6 +172,7 @@ def load_from_yaml(  # type: ignore
             pipeline_config=pipeline_config,
             pipeline_name=pipeline_name,
             overwrite_with_env_variables=overwrite_with_env_variables,
+            strict_version_check=strict_version_check,
             address=address,
             ray_args=ray_args,
         )


Test patch:
diff --git a/test/pipelines/test_ray.py b/test/pipelines/test_ray.py
--- a/test/pipelines/test_ray.py
+++ b/test/pipelines/test_ray.py
@@ -23,7 +23,7 @@ def shutdown_ray():
 @pytest.mark.parametrize("document_store_with_docs", ["elasticsearch"], indirect=True)
 def test_load_pipeline(document_store_with_docs):
     pipeline = RayPipeline.load_from_yaml(
-        SAMPLES_PATH / "pipeline" / "test_ray_pipeline.yaml",
+        SAMPLES_PATH / "pipeline" / "ray.haystack-pipeline.yml",
         pipeline_name="ray_query_pipeline",
         ray_args={"num_cpus": 8},
     )
diff --git a/test/samples/pipeline/test_ray_pipeline.yaml b/test/samples/pipeline/ray.haystack-pipeline.yml
similarity index 97%
rename from test/samples/pipeline/test_ray_pipeline.yaml
rename to test/samples/pipeline/ray.haystack-pipeline.yml
--- a/test/samples/pipeline/test_ray_pipeline.yaml
+++ b/test/samples/pipeline/ray.haystack-pipeline.yml
@@ -1,4 +1,5 @@
 version: ignore
+extras: ray
 
 components:
   - name: DocumentStore
@@ -37,7 +38,6 @@ components:
 
 pipelines:
   - name: ray_query_pipeline
-    type: RayPipeline
     nodes:
       - name: ESRetriever
         replicas: 2