Repository: meltano/meltano

Issue:
[Bug]: Snowplow tracker does not timeout when endpoint is unavailable
### Meltano Version

2.3.0

### Python Version

NA

### Bug scope

Other

### Operating System

NA

### Description

First reported by @dingobar here: https://meltano.slack.com/archives/C01TCRBBJD7/p1659099832402809

> Hi all, thanks for a great product! Everything is working great locally, and we've now deployed our meltano project on kubernetes where we already had airflow running. However, there is a hiccup: The job hangs after Environment 'dev' is active (see screenshot). 
> ![image](https://user-images.githubusercontent.com/11428666/181815830-029559ca-a21a-484e-8030-1fe9e5517584.png)
> 
> After about 15 minutes or so, the actual task completes, but the container keeps hanging telling me that the meltano run process has not yet exited and is blocked somehow. To me it seems like it is either stuck in an infinite loop, or trying to call something over the network with (seemingly) infinite retries/timeout.
> 
> We have a strict firewall policy where we explicitly name the domains to be whitelisted for egress out from our environment (required as we are in the telco business with very strict data protection requirements). My theory is that meltano is trying to call out to some external site. I have disabled telemetry in meltano in general (MELTANO_SEND_ANONYMOUS_USAGE_STATS="False" as stated in the docs).
> 
> As evident in the logs, it hangs for about 6 minutes before getting started:
> ![image](https://user-images.githubusercontent.com/11428666/181815889-85030c26-3cf8-4e7d-8116-24fad9ec05ad.png)
> 
> Update: So the culprit is found. It is trying to contact "[sp.meltano.com](http://sp.meltano.com/)" with an exponential retry backoff. How can I disable this behavior?
> ![image](https://user-images.githubusercontent.com/11428666/181815966-ea69a683-d853-42a8-84b9-fa5ae707069c.png)


We should:
- Change the default `request_timeout` value for `meltano.core.tracking.Tracker` from `None` to `3.5`.
- Add a test that simulates the endpoint not responding.

### Code

_No response_


Discussion:
@WillDaSilva thanks for opening and picking up!
I attempted to use `requests_mock` (which we already have as a dependency) to mock the timeout, but unfortunately when it is being used the `requests` library doesn't seem to raise a timeout error. Hosting a basic HTTP server in a subprocess that accepts the connection but never responds to GET/POST requests might be the way to go.

Resolving patch:
diff --git a/src/meltano/core/tracking/tracker.py b/src/meltano/core/tracking/tracker.py
--- a/src/meltano/core/tracking/tracker.py
+++ b/src/meltano/core/tracking/tracker.py
@@ -85,13 +85,16 @@ class Tracker:  # noqa: WPS214 - too many methods 16 > 15
     def __init__(
         self,
         project: Project,
-        request_timeout: float | tuple[float, float] | None = None,
+        request_timeout: float | tuple[float, float] | None = 3.5,
     ):
         """Initialize a tracker for the Meltano project.
 
         Args:
             project: The Meltano project.
-            request_timeout: Timeout for the HTTP requests. Can be set either as single float value which applies to both `connect` AND `read` timeout, or as tuple with two float values which specify the `connect` and `read` timeouts separately.
+            request_timeout: Timeout for the HTTP requests. Can be set either
+                as single float value which applies to both `connect` and
+                `read` timeout, or as tuple with two float values which specify
+                the `connect` and `read` timeouts separately.
         """
         self.project = project
         self.settings_service = ProjectSettingsService(project)


Test patch:
diff --git a/tests/meltano/core/tracking/test_tracker.py b/tests/meltano/core/tracking/test_tracker.py
--- a/tests/meltano/core/tracking/test_tracker.py
+++ b/tests/meltano/core/tracking/test_tracker.py
@@ -6,6 +6,9 @@
 import subprocess
 import uuid
 from contextlib import contextmanager
+from http import server as server_lib
+from threading import Thread
+from time import sleep
 from typing import TYPE_CHECKING, Any
 
 import mock
@@ -13,6 +16,7 @@
 
 from meltano.core.project import Project
 from meltano.core.project_settings_service import ProjectSettingsService
+from meltano.core.tracking.contexts.cli import CliEvent
 from meltano.core.tracking.contexts.environment import EnvironmentContext
 from meltano.core.tracking.contexts.exception import ExceptionContext
 from meltano.core.tracking.contexts.project import ProjectContext
@@ -343,3 +347,58 @@ def track_unstruct_event(self, _, contexts):
             "send_anonymous_usage_stats", False, True
         )
         assert passed
+
+    @pytest.mark.parametrize(  # noqa: WPS317
+        ("sleep_duration", "timeout_should_occur"),
+        ((1.0, False), (5.0, True)),
+        ids=("no_timeout", "timeout"),
+    )
+    def test_timeout_if_endpoint_unavailable(
+        self, project: Project, sleep_duration, timeout_should_occur
+    ):
+        """Test to ensure that the default tracker timeout is respected.
+
+        An HTTP sever is run on another thread, which handles request sent from
+        the Snowplow tracker. When `timeout_should_occur` is `False`, we check
+        that this sever is properly masqerading as a Snowplow endpoint. When
+        `timeout_should_occur` is `True`, we check that when the server takes
+        too long to respond, we timeout and continue without raising an error.
+        """
+        timeout_occured = False
+
+        class HTTPRequestHandler(server_lib.SimpleHTTPRequestHandler):
+            def do_GET(self) -> None:  # noqa: N802
+                sleep(sleep_duration)
+                self.send_response(200, "OK")
+                self.end_headers()
+
+        server = server_lib.HTTPServer(("localhost", 0), HTTPRequestHandler)
+        server_thread = Thread(
+            target=server.serve_forever, kwargs={"poll_interval": 0.1}
+        )
+        server_thread.start()
+
+        ProjectSettingsService(project).set(
+            "snowplow.collector_endpoints",
+            f'["http://localhost:{server.server_port}"]',
+        )
+
+        def emitter_failure_callback(_, failure_events: list):
+            nonlocal timeout_occured
+            # `timeout_occured` is technically a misnomer here, since there are
+            # multiple reasons for failure, but this callback doesn't let us
+            # distinguish. We can rely on the `timeout_should_occur is True`
+            # case to ensure that this callback won't be called for non-timeout
+            # reasons.
+            timeout_occured = True
+
+        tracker = Tracker(project)
+        assert len(tracker.snowplow_tracker.emitters) == 1
+        tracker.snowplow_tracker.emitters[0].on_failure = emitter_failure_callback
+
+        tracker.track_command_event(CliEvent.started)
+
+        server.shutdown()
+        server_thread.join()
+
+        assert timeout_occured is timeout_should_occur