Repository: deepset-ai/haystack

Issue:
503 error message shown if posthog is unavailable
**Describe the bug**
Haystack sends telemetry events to a posthog server but when the server is unavailable it shows an error message. There is nothing the user can do about that so we should hide that error message.

Quick workaround is to either [disable telemetry](https://docs.haystack.deepset.ai/docs/telemetry#how-can-i-opt-out) or set the log level of posthog to `CRITICAL` in the application code:
```python
import logging
logging.getLogger("posthog").setLevel(logging.CRITICAL)
```

another workaround is
```python
import logging
logging.raiseExceptions = False
```

There is only [one point in the code](https://github.com/deepset-ai/haystack/blob/main/haystack/telemetry.py#L156), where we send telemetry events via posthog and it is wrapped in a try-catch block. It seems to me that this issue is just about hiding a log message. We have a line of code that is meant to limit posthog logging to level `CRITICAL` but it doesn't work as intended: https://github.com/deepset-ai/haystack/blob/fa17f0973e884c41ddff32d955aacb9837c0abfd/haystack/telemetry.py#L34
Maybe because no handler is specified. I'll investigate this further.

**Error message**
The execution of the program continues as expected without failing. The following error message is shown:
```python
--- Logging error ---
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/request.py", line 57, in _process_response
    payload = res.json()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/consumer.py", line 78, in upload
    self.request(batch)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/consumer.py", line 138, in request
    send_request()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/backoff/_sync.py", line 94, in retry
    ret = target(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/consumer.py", line 136, in send_request
    batch_post(self.api_key, self.host, gzip=self.gzip, timeout=self.timeout, batch=batch)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/request.py", line 75, in batch_post
    return _process_response(res, success_message="data uploaded successfully", return_json=False)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/request.py", line 61, in _process_response
    raise APIError(res.status_code, res.text)
posthog.request.APIError: [PostHog] <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>
 (503)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/consumer.py", line 62, in run
    self.upload()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/py39_dec/lib/python3.9/site-packages/posthog/consumer.py", line 81, in upload
    self.log.error("error uploading: %s", e)
Message: 'error uploading: %s'
Arguments: (APIError(503, '<html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n</body>\r\n</html>\r\n'),)

```

**Expected behavior**
No error message is shown to the user and posthog silently fails to send a telemetry event. Haystack continues with the code execution.

**To Reproduce**
Initialize any Haystack node while https://tm.hs.deepset.ai/ returns 503 or use Haystack offline. For example:
```python
from haystack.document_stores import InMemoryDocumentStore
document_store = InMemoryDocumentStore(use_bm25=True)
```
results in the printed message
```
>>> ERROR:posthog:error uploading: [PostHog] <html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
</body>
</html>
 (503)
```


Discussion:

Resolving patch:
diff --git a/haystack/telemetry.py b/haystack/telemetry.py
--- a/haystack/telemetry.py
+++ b/haystack/telemetry.py
@@ -31,8 +31,11 @@
 logger = logging.getLogger(__name__)
 
 # disable posthog logging
-logging.getLogger("posthog").setLevel(CRITICAL)
-logging.getLogger("backoff").setLevel(CRITICAL)
+for module_name in ["posthog", "backoff"]:
+    logging.getLogger(module_name).setLevel(CRITICAL)
+    # Prevent module from sending errors to stderr when an exception is encountered during an emit() call
+    logging.getLogger(module_name).addHandler(logging.NullHandler())
+    logging.getLogger(module_name).propagate = False
 
 
 class TelemetryFileType(Enum):


Test patch: