Repository: meltano/meltano

Issue:
bug: Test connector - asyncio
### Meltano Version

2.4.0

### Python Version

3.9

### Bug scope

CLI (options, error messages, logging, etc.)

### Operating System

Rocky Linux release 8.6

### Description

```
meltano config tap-postgres test
```

above command works fine till version 2.1.0 and python 3.9.7.

But it fails when i switched to 2.2.0,2.3.0 and 2.4.0

```
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x7fcad8ae8670>
Traceback (most recent call last):
  File "/usr/lib64/python3.9/asyncio/base_subprocess.py", line 126, in __del__
    self.close()
  File "/usr/lib64/python3.9/asyncio/base_subprocess.py", line 104, in close
    proto.pipe.close()
  File "/usr/lib64/python3.9/asyncio/unix_events.py", line 536, in close
    self._close(None)
  File "/usr/lib64/python3.9/asyncio/unix_events.py", line 560, in _close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 746, in call_soon
    self._check_closed()
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
```

To fix it , I changed file in `cli/config.y`:

```diff
-- is_valid, detail = asyncio.run(_validate()) (removed)
++is_valid, detail = asyncio.new_event_loop().run_until_complete(_validate())  (added)
```

Kindly get it check at your end and fix it in coming releases.

### Code

To fix it, I changed file in `cli/config.py`:

```diff
- is_valid, detail = asyncio.run(_validate())
+ is_valid, detail = asyncio.new_event_loop().run_until_complete(_validate())
```



Discussion:
@pandemicsyn @edgarrmondragon can you validate this bug and potential fix?

@AvinashGupta802 perhaps a PR for this would be better if you're willing to submit one
@AvinashGupta802 @tayloramurphy I can't reproduce this on Meltano 2.4.0, macOS and Python 3.9.7, and Ubuntu with the latest Python 3.9 (3.9.13) doesn't seem to have this issue.

I'd suggest upgrading to the latest Python 3.9.x, because I do see some asyncio bug fixes in the changelog: https://docs.python.org/3.9/whatsnew/changelog.html#python-3-9-8-final
I tried with Python 3.9.13 same error. Screenshot attached. Post change in code it works. Screenshot attached
![image](https://user-images.githubusercontent.com/68946032/184810352-4f082e6a-33ea-44e9-98d9-3db0997c4660.png)
![image](https://user-images.githubusercontent.com/68946032/184810400-e513e07f-3d8f-4d9e-9f71-fe07f34c95e8.png)
![image](https://user-images.githubusercontent.com/68946032/184810410-c7ab1785-7d7e-4d20-9194-d0de61dcfe6c.png)


@AvinashGupta802 I'm curious, are you seeing similar issues with other commands such as  `meltano invoke` or `meltano test` ? 


invoke command works fine. Issue only with test command
It looks like this is most likely related to https://github.com/python/cpython/issues/88050. The test/validation actually **always** terminate the spawned subprocess early:

https://github.com/meltano/meltano/blob/ecac36b7d1094955e3ba2c7fd98d4f4201c4fe25/src/meltano/core/plugin_test_service.py#L70-L75

This error could also trigger on `meltano run` or `meltano elt` invocations where a plugin is being terminated early (e.g. a target fails, and we kill the tap). 

Long term there's a pending fix upstream at https://github.com/python/cpython/pull/32073 but I'm not sure if/when that will actually land. So does feel like a temp fix/work around on our end might be warranted. 

Either the fix that @AvinashGupta802 proposed (probably ok for `test`, but not sure at a glance what the implications would be for switching `run`/`elt` to this), or rolling with one of the workarounds listed in the python bug report. Namely, from https://github.com/python/cpython/issues/88050#issuecomment-1093911484 and adding a call to `process._transport.close()`

/cc @edgarrmondragon @WillDaSilva (be curious what y'all think and if this actually what's happening or if theres a simpler issue at play). 
@pandemicsyn @WillDaSilva `process._transport.close()` seems reasonably stable and might be good enough. I guess the options then are

- do we wait for the current state of things to cause more issues for users because of the underlying bug or
- do we use the "protected" API and then wait for that to break with some future Python release

?
Since this is sort of an edge case - personally, i think I'd be ok with using the protected call and trying to keep tabs on that issue to resolve upstream. I'm guessing that's not getting back ported 3.7. This also isn't our first run-in with quirky asyncio.Subprocess behavior. We also had:

https://gitlab.com/meltano/meltano/-/issues/3347#note_894964904

@edgarrmondragon I guess there's also a sort of middle ground. Since this seems more likely to happen in `meltano test` where we always terminate the subprocess, we could maybe only implement the workaround of using the projected API there. 
> @edgarrmondragon I guess there's also a sort of middle ground. Since this seems more likely to happen in `meltano test` where we always terminate the subprocess, we could maybe only implement the workaround of using the projected API there.

@pandemicsyn that works for me
@AvinashGupta802 if you would like to take a stab at fixing this we'd definitely be open to a PR that implements the fix from https://github.com/python/cpython/issues/88050#issuecomment-1093911484 for https://github.com/meltano/meltano/blob/ecac36b7d1094955e3ba2c7fd98d4f4201c4fe25/src/meltano/core/plugin_test_service.py#L70-L75. 

@tayloramurphy if this goes on the backlog, realistically it's probably a 2, but I assigned a 4 in case some unexpected side effects pop up. 
@pandemicsyn ,  I have already informed with code changes done at my end  to resolve the issue faced by me.

Resolving patch:
diff --git a/src/meltano/core/plugin_invoker.py b/src/meltano/core/plugin_invoker.py
--- a/src/meltano/core/plugin_invoker.py
+++ b/src/meltano/core/plugin_invoker.py
@@ -433,7 +433,7 @@ async def _invoke(
                     self.plugin, self.plugin.executable
                 ) from err
 
-    async def invoke_async(self, *args, **kwargs):
+    async def invoke_async(self, *args, **kwargs) -> asyncio.subprocess.Process:
         """Invoke a command.
 
         Args:


Test patch:
diff --git a/src/meltano/core/plugin_test_service.py b/src/meltano/core/plugin_test_service.py
--- a/src/meltano/core/plugin_test_service.py
+++ b/src/meltano/core/plugin_test_service.py
@@ -18,12 +18,24 @@ class PluginTestServiceFactory:
     """Factory class to resolve a plugin test service."""
 
     def __init__(self, plugin_invoker: PluginInvoker):
-        """Construct a PluginTestServiceFactory instance."""
+        """Construct a PluginTestServiceFactory instance.
+
+        Args:
+            plugin_invoker: The invocation instance of the plugin to test.
+        """
         self.plugin_invoker = plugin_invoker
 
     def get_test_service(self):
-        """Resolve a test service instance for a plugin type."""
+        """Resolve a test service instance for a plugin type.
+
+        Returns:
+            The test service instance.
+
+        Raises:
+            PluginNotSupportedError: If the plugin type is not supported for testing.
+        """
         test_services = {PluginType.EXTRACTORS: ExtractorTestService}
+
         try:
             return test_services[self.plugin_invoker.plugin.type](self.plugin_invoker)
         except KeyError as err:
@@ -34,19 +46,27 @@ class PluginTestService(ABC):
     """Abstract base class for plugin test operations."""
 
     def __init__(self, plugin_invoker: PluginInvoker):
-        """Construct a PluginTestService instance."""
+        """Construct a PluginTestService instance.
+
+        Args:
+            plugin_invoker: The invocation instance of the plugin to test
+        """
         self.plugin_invoker = plugin_invoker
 
     @abstractmethod
-    def validate(self) -> bool | str:
+    async def validate(self) -> tuple[bool, str]:
         """Abstract method to validate plugin configuration."""
 
 
 class ExtractorTestService(PluginTestService):
     """Handle extractor test operations."""
 
-    async def validate(self) -> bool | str:
-        """Validate extractor configuration."""
+    async def validate(self) -> tuple[bool, str]:
+        """Validate extractor configuration.
+
+        Returns:
+            The validation result and supporting context message (if applicable).
+        """
         process = None
 
         try:
@@ -71,8 +91,13 @@ async def validate(self) -> bool | str:
 
             if message_type == "RECORD":
                 process.terminate()
-                return True, None
+                break
 
-        await process.wait()
+        returncode = await process.wait()
 
-        return False, last_line if process.returncode else "No RECORD message received"
+        # considered valid if subprocess is terminated (exit status < 0) on RECORD message received
+        # see https://docs.python.org/3/library/subprocess.html#subprocess.CompletedProcess.returncode
+        return (
+            returncode < 0,
+            last_line if returncode else "No RECORD message received",
+        )
diff --git a/tests/meltano/api/controllers/test_orchestration.py b/tests/meltano/api/controllers/test_orchestration.py
--- a/tests/meltano/api/controllers/test_orchestration.py
+++ b/tests/meltano/api/controllers/test_orchestration.py
@@ -133,8 +133,8 @@ def test_test_plugin_configuration_success(
         mock_invoke = mock.Mock()
         mock_invoke.sterr.at_eof.side_effect = True
         mock_invoke.stdout.at_eof.side_effect = (False, True)
-        mock_invoke.wait = AsyncMock(return_value=0)
-        mock_invoke.returncode = 0
+        mock_invoke.wait = AsyncMock(return_value=-1)
+        mock_invoke.returncode = -1
         payload = json.dumps({"type": "RECORD"}).encode()
         mock_invoke.stdout.readline = AsyncMock(return_value=b"%b" % payload)
 
diff --git a/tests/meltano/cli/test_config.py b/tests/meltano/cli/test_config.py
--- a/tests/meltano/cli/test_config.py
+++ b/tests/meltano/cli/test_config.py
@@ -78,8 +78,8 @@ def test_config_test(
         mock_invoke = mock.Mock()
         mock_invoke.sterr.at_eof.side_effect = True
         mock_invoke.stdout.at_eof.side_effect = (False, True)
-        mock_invoke.wait = AsyncMock(return_value=0)
-        mock_invoke.returncode = 0
+        mock_invoke.wait = AsyncMock(return_value=-1)
+        mock_invoke.returncode = -1
         payload = json.dumps({"type": "RECORD"}).encode()
         mock_invoke.stdout.readline = AsyncMock(return_value=b"%b" % payload)
 
diff --git a/tests/meltano/core/test_plugins_test_service.py b/tests/meltano/core/test_plugin_test_service.py
similarity index 95%
rename from tests/meltano/core/test_plugins_test_service.py
rename to tests/meltano/core/test_plugin_test_service.py
--- a/tests/meltano/core/test_plugins_test_service.py
+++ b/tests/meltano/core/test_plugin_test_service.py
@@ -45,8 +45,8 @@ class TestExtractorTestService:
     def setup(self, mock_invoker):
         self.mock_invoke = Mock()
         self.mock_invoke.name = "utility-mock"
-        self.mock_invoke.wait = AsyncMock(return_value=0)
-        self.mock_invoke.returncode = 0
+        self.mock_invoke.wait = AsyncMock(return_value=-1)
+        self.mock_invoke.returncode = -1
         self.mock_invoker = mock_invoker
         self.mock_invoker.invoke_async = AsyncMock(return_value=self.mock_invoke)
 
@@ -61,7 +61,7 @@ async def test_validate_success(self):
         is_valid, detail = await ExtractorTestService(self.mock_invoker).validate()
 
         assert is_valid
-        assert detail is None
+        assert detail == MOCK_RECORD_MESSAGE
 
     @pytest.mark.asyncio
     async def test_validate_success_ignore_non_json(self):
@@ -74,7 +74,7 @@ async def test_validate_success_ignore_non_json(self):
         is_valid, detail = await ExtractorTestService(self.mock_invoker).validate()
 
         assert is_valid
-        assert detail is None
+        assert detail == MOCK_RECORD_MESSAGE
 
     @pytest.mark.asyncio
     async def test_validate_success_ignore_non_record_msg(self):
@@ -90,7 +90,7 @@ async def test_validate_success_ignore_non_record_msg(self):
         is_valid, detail = await ExtractorTestService(self.mock_invoker).validate()
 
         assert is_valid
-        assert detail is None
+        assert detail == MOCK_RECORD_MESSAGE
 
     @pytest.mark.asyncio
     async def test_validate_success_stop_after_record_msg(self):
@@ -107,7 +107,7 @@ async def test_validate_success_stop_after_record_msg(self):
         is_valid, detail = await ExtractorTestService(self.mock_invoker).validate()
 
         assert is_valid
-        assert detail is None
+        assert detail == MOCK_RECORD_MESSAGE
 
         assert self.mock_invoke.stdout.readline.call_count == 2
 
@@ -119,6 +119,9 @@ async def test_validate_failure_no_record_msg(self):
             return_value=(b"%b" % MOCK_STATE_MESSAGE.encode())
         )
 
+        self.mock_invoke.wait = AsyncMock(return_value=0)
+        self.mock_invoke.returncode = 0
+
         is_valid, detail = await ExtractorTestService(self.mock_invoker).validate()
 
         assert not is_valid