Repository: deepset-ai/haystack

Issue:
Bug: `RayPipeline` event sending fails crashing the app even with telemetry disabled
**Describe the bug**
`RayPipeline` event sending (for telemetry) fails and crashing the app even with telemetry disabled.

This is because
(1) Event sending is called and pipeline components are serialized even when telemetry is disabled. Telemetry being disabled only will stop the actual sending out of the data, while the serialization still happens (even if it is not sent in the end)
(2) The pipeline serialization ([`get_config()`](https://github.com/deepset-ai/haystack/blob/main/haystack/pipelines/base.py#L2049)) can't handle the components of `RayPipeline`, so it crashes the app once it is called.

This is a nasty bug, as it will only becomes visible once the app has received enough calls to trigger the event sending - so you might not see this in dev or maybe even in testing, possible only in prod once the app has received enough calls to trigger its telemetry even sending (even with telemetry disabled).

As this issue is related to not being able to serialize the `RayPipeline` components, this should only affect those who use `RayPipelines`.

**Error message**
```
File "/app_home/.local/share/virtualenvs/app-Fi1MU8X9/src/farm-haystack/haystack/pipelines/ray.py", line 404, in run
self.send_pipeline_event_if_needed(is_indexing=file_paths is not None)
| | -> None
| -> <function Pipeline.send_pipeline_event_if_needed at 0x7ff4de281cf0>
-> <haystack.pipelines.ray.RayPipeline object at 0x7ff5995b66e0>

File "/app_home/.local/share/virtualenvs/app-Fi1MU8X9/src/farm-haystack/haystack/pipelines/base.py", line 2314, in send_pipeline_event_if_needed
self.send_pipeline_event(is_indexing)
| | -> False
| -> <function Pipeline.send_pipeline_event at 0x7ff4de281c60>
-> <haystack.pipelines.ray.RayPipeline object at 0x7ff5995b66e0>

File "/app_home/.local/share/virtualenvs/app-Fi1MU8X9/src/farm-haystack/haystack/pipelines/base.py", line 2295, in send_pipeline_event
json_repr = json.dumps(self.get_config(), sort_keys=True, default=lambda o: "<not serializable>")
| | | -> <function Pipeline.get_config at 0x7ff4de2816c0>
| | -> <haystack.pipelines.ray.RayPipeline object at 0x7ff5995b66e0>
| -> <function dumps at 0x7ff59e88fd90>
-> <module 'json' from '/usr/lib/python3.10/json/__init__.py'>

File "/app_home/.local/share/virtualenvs/app-Fi1MU8X9/src/farm-haystack/haystack/pipelines/base.py", line 2069, in get_config
raise PipelineError(f"Component name '{component.name}' does not match node name '{node_name}'.")
-> <class 'haystack.errors.PipelineError'>

haystack.errors.PipelineError: Component name 'RayServeSyncHandle(deployment='prompt-node-llm-demo')' does not match node name 'prompt-node-llm-demo'.
```

Even if you fix this particular error thrown here, there will be additional ones due to the RayPipeline components are not serializable with the methods called from `get_config()`. A complete rework would be required to weed out all of them.

**Expected behavior**
For the send method to work on RayPipeline the whole sending should be rewritten to ensure that it works with the RayPipeline.

Alternatively - as this is a bug affecting all RayPipeline deployments without a workaround, so it is urgent to provide a workaround - sending could just be disabled for RayPipeline deployments for now. I will make a PR for this "workaround".

**Additional context**
As I have noted above, having telemetry disabled has no effect on this, due to the somewhat strange implementation of disabling telemetry, which still collects ALL data to be sent to telemetry, but due to it being disabled it is not sending it out in the end.
It might be worth refactoring that code to eliminate the event collection completely when telemetry is disabled by the user.
This could have allowed the users affected by this to have a workaround (simply by disabling telemetry) and could save computation cycles for those who have the telemetry disabled and serializing for event sending is wasteful.

**To Reproduce**
Create any app with `RayPipelines` and generate enough calls against it to trigger the event sending (or just disable [this check](https://github.com/deepset-ai/haystack/blob/main/haystack/pipelines/base.py#L2313) to trigger it immediately without waiting to have enough calls accumulated)

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Any 
 - GPU/CPU: Any
 - Haystack version (commit or version number): Latest dev from GitHub
 - DocumentStore: Any
 - Reader: Any
 - Retriever: Any



Discussion:

Resolving patch:
diff --git a/haystack/pipelines/base.py b/haystack/pipelines/base.py
--- a/haystack/pipelines/base.py
+++ b/haystack/pipelines/base.py
@@ -51,7 +51,7 @@
 from haystack.nodes.base import BaseComponent, RootNode
 from haystack.nodes.retriever.base import BaseRetriever
 from haystack.document_stores.base import BaseDocumentStore
-from haystack.telemetry import send_event, send_custom_event
+from haystack.telemetry import send_event, send_custom_event, is_telemetry_enabled
 from haystack.utils.experiment_tracking import MLflowTrackingHead, Tracker as tracker
 
 
@@ -2309,18 +2309,21 @@ def send_pipeline_event(self, is_indexing: bool = False):
         self.last_window_run_total = self.run_total
 
     def send_pipeline_event_if_needed(self, is_indexing: bool = False):
-        should_send_event = self.has_event_time_interval_exceeded() or self.has_event_run_total_threshold_exceeded()
-        if should_send_event and not self.sent_event_in_window:
-            self.send_pipeline_event(is_indexing)
-            self.sent_event_in_window = True
-        elif self.has_event_time_interval_exceeded():
-            self.sent_event_in_window = False
-
-    def has_event_time_interval_exceeded(self):
+        if is_telemetry_enabled():
+            should_send_event = (
+                self._has_event_time_interval_exceeded() or self._has_event_run_total_threshold_exceeded()
+            )
+            if should_send_event and not self.sent_event_in_window:
+                self.send_pipeline_event(is_indexing)
+                self.sent_event_in_window = True
+            elif self._has_event_time_interval_exceeded():
+                self.sent_event_in_window = False
+
+    def _has_event_time_interval_exceeded(self):
         now = datetime.datetime.now(datetime.timezone.utc)
         return now - self.time_of_last_sent_event > self.event_time_interval
 
-    def has_event_run_total_threshold_exceeded(self):
+    def _has_event_run_total_threshold_exceeded(self):
         return self.run_total - self.last_window_run_total > self.event_run_total_threshold
 
 
diff --git a/haystack/pipelines/ray.py b/haystack/pipelines/ray.py
--- a/haystack/pipelines/ray.py
+++ b/haystack/pipelines/ray.py
@@ -401,7 +401,8 @@ async def run_async(  # type: ignore
                 i += 1  # attempt executing next node in the queue as current `node_id` has unprocessed predecessors
 
         self.run_total += 1
-        self.send_pipeline_event_if_needed(is_indexing=file_paths is not None)
+        # Disabled due to issue https://github.com/deepset-ai/haystack/issues/3970
+        # self.send_pipeline_event_if_needed(is_indexing=file_paths is not None)
         return node_output
 
 


Test patch: