Repository: meltano/meltano

Issue:
Intermittent errors when doing meltano config airflow
Can't seem to consistently reproduce this.

```bash
❯ meltano config airflow set webserver web_server_port 8081
2022-06-09T02:07:03.866027Z [info     ] Environment 'dev' is active
[Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'
```

but if I run `meltano invoke airflow:ui` and then kill the process once it's trying to connect I can successfully run 

```bash
❯ meltano config airflow set webserver web_server_port 8081
2022-06-09T02:09:10.771279Z [info     ] Environment 'dev' is active
Orchestrator 'airflow' setting 'webserver.web_server_port' was set in the active environment in `meltano.yml`: 8081
```

so clearly something about the airflow.cfg being present or not. 

I would expect to be able to set the config in my meltano.yml even if the airflow.cfg is not present.

It seems like it's something with invoke because this is what happens if I try to change the port to 8082, then do an invoke, and then reset the port again

```bash
❯ meltano config airflow set webserver web_server_port 8082
2022-06-09T02:10:32.255003Z [info     ] Environment 'dev' is active
Orchestrator 'airflow' setting 'webserver.web_server_port' was set in the active environment in `meltano.yml`: 8082
❯ meltano invoke airflow version
2022-06-09T02:10:47.148948Z [info     ] Environment 'dev' is active
2.1.2
❯ meltano config airflow set webserver web_server_port 8082
2022-06-09T02:11:00.416107Z [info     ] Environment 'dev' is active
[Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'
```


Discussion:
@tayloramurphy Can you run again with `--log-level=debug` and share the full stacktrace? I'm not sure why `meltano config meltano set` would use the actual Airflow config file (`.meltano/run/airflow/airflow.cfg`), that should only be relevant on invoke.
@DouweM 

`meltano --log-level=debug --no-environment config airflow set webserver web_server_port 8081`

```
2022-06-09T02:17:06.634275Z [info     ] No environment is active
2022-06-09T02:17:06.678497Z [debug    ] Creating engine <meltano.core.project.Project object at 0x1038143a0>@sqlite:////Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/meltano.db
2022-06-09T02:17:06.778593Z [debug    ] Found plugin parent            parent=airflow plugin=airflow source=lockfile
2022-06-09T02:17:06.985625Z [debug    ] Using selector: KqueueSelector
2022-06-09T02:17:07.009197Z [debug    ] Invoking: ['/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/orchestrators/airflow/venv/bin/airflow', '--help']
<...envstuff...>
2022-06-09T02:17:07.680278Z [debug    ] Generated default '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'
2022-06-09T02:17:07.683015Z [debug    ] [Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg' 
Traceback (most recent call last):
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/cli/__init__.py", line 51, in main
    cli(obj={"project": None})
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/core.py", line 1256, in invoke
    Command.invoke(self, ctx)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/cli/params.py", line 23, in decorate
    return func(*args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/cli/params.py", line 56, in decorate
    func(project, *args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/cli/config.py", line 81, in config
    run_async(invoker.prepare(session))
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/utils/__init__.py", line 52, in run_async
    loop.run_until_complete(future)
  File "/Users/taylormurphy/.pyenv/versions/3.9.7/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/plugin_invoker.py", line 223, in prepare
    async with self.plugin.trigger_hooks("configure", self, session):
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks
    await self.__class__.trigger(self, f"before_{hook_name}", *args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger
    raise err
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger
    await hook_func(target, *args, **kwargs)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/plugin/airflow.py", line 126, in before_configure
    self.update_config_file(invoker)
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/core/plugin/airflow.py", line 77, in update_config_file
    with airflow_cfg_path.open() as airflow_cfg_file_to_read:
  File "/Users/taylormurphy/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/Users/taylormurphy/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/taylormurphy/.local/pipx/venvs/meltano/lib/python3.9/site-packages/meltano/cli/__init__.py", line 59, in main
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: [Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'
[Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'
```
@tayloramurphy OK, so the offending line is https://github.com/meltano/meltano/blob/main/src/meltano/cli/config.py#L81, added in https://github.com/meltano/meltano/commit/ef0151da69228c282b0785b2073e14e4da6c802a. Let me try to find out why that would be needed...
@pandemicsyn Do you remember why that was added there, and not just in the `meltano config <plugin> test` entrypoint? `PluginInvoker.prepare` is a pretty expensive method that triggers configuration hooks: https://github.com/meltano/meltano/blob/main/src/meltano/core/plugin_invoker.py#L223, which in the case of Airflow go as far as invoking the plugin and building its database, which shouldn't be necessary if we're just updating Meltano's conception of its config. Even with `test`, I'm not sure we need that extra call as it already uses `PluginInvoker.prepared` which does that `prepare` call as well: https://github.com/meltano/meltano/blob/main/src/meltano/cli/config.py#L286.

Note that this doesn't explain why `airflow.cfg` doesn't exist inside the Airfow `before_configure` hook, as it starts by creating it by [calling `airflow --help`](https://github.com/meltano/meltano/blob/main/src/meltano/core/plugin/airflow.py#L113). But at least this would make `meltano config` not be affected by that.
@tayloramurphy I'm trying to figure out why `Airflow.before_configure` would be failing when it's called through `meltano config`... 

Do you ever see `meltano invoke airflow version` fail with the same error? Or only ever `meltano config`? 

When you saw `meltano config` fail, were you running `meltano invoke airflow ...` in another terminal session?
@DouweM haven't seen `meltano invoke airflow version` fail. Only config. 

That commit rings a bell about some work @edgarrmondragon did around `meltano test` and some bugs we had with Airflow I think. @edgarrmondragon do you recall?

https://github.com/meltano/meltano/issues/3159 might be the issue I was talking about
@tayloramurphy I recall that, though it was specific to `meltano test` and its eagerness to "prepare" an invoke context for all plugins: https://gitlab.com/meltano/meltano/-/issues/3233#note_851804821, which as @DouweM mentioned is an operation with lots of side effects.

So this seems to have a similar cause, but in a different place in the code.
> @pandemicsyn Do you remember why that was added there, and not just in the meltano config <plugin> test entrypoint? PluginInvoker.prepare is a pretty expensive method that triggers configuration hooks: https://github.com/meltano/meltano/blob/main/src/meltano/core/plugin_invoker.py#L223, which in the case of Airflow go as far as invoking the plugin and building its database, which shouldn't be necessary if we're just updating Meltano's conception of its config. Even with test, I'm not sure we need that extra call as it already uses PluginInvoker.prepared which does that prepare call as well: https://github.com/meltano/meltano/blob/main/src/meltano/cli/config.py#L286.

@DouweM I honestly can't recall,  the original MR is here https://gitlab.com/meltano/meltano/-/merge_requests/2179  , I took it over because I caused a bunch of conflicts with that big async refactor. I probably didn't know any better.
~~errr  https://gitlab.com/meltano/meltano/-/merge_requests/2326/diffs isn't actually where that came from,  neither is https://gitlab.com/meltano/meltano/-/merge_requests/2179 which are the two MR's linked in the issue for the changelog entry in  https://github.com/meltano/meltano/commit/ef0151da69228c282b0785b2073e14e4da6c802a~~

ingore me.
@pandemicsyn OK, I think we can at least fix the `meltano config` issue then by losing that superfluous `invoke` call. If it's never happened on `meltano invoke` we don't really need to dive deeper into that part of it.

I'll put this in the project/board, you can figure out with AJ if it makes sense for you to pick it up or someone else should!
@pandemicsyn - do you think you have a path forward here on a fix, either that you could implement or that you could guide another engineer thought?

If so, I'm in favor of picking this up in next iteration.
@aaronsteers I think the trick might be trying to figure out why it was included in the first place, some what worried that theres a non obvious reason why it was included when we added `meltano test`.  Though its very likely I just didn't know any better at the time and was just trying to make the async refactor play nice with the conflicting `meltano test` MR.

Happy to pick this up, but i might need you to clarify my priority/order - with the other currently assigned issues I doubt I can deliver all three.


@aaronsteers So, yea, this might actually be a really quick fix. I was worried I had added this intentionally for a reason I can't recall, but I think this just got introduced accidentally when we picked up the community contrib: 

https://gitlab.com/meltano/meltano/-/merge_requests/2179/diffs?commit_id=7bad854e3d7ee4c4b2a0ebede24e1ee41797d3b1#bd50d05c414b847ffb1be734807753e08662c778_52_58 

And then I missed the implications of that addition being in that top-level click group, and just blindly made it async aware to resolve the merge conflict from having just landed that large async refactor:

https://gitlab.com/meltano/meltano/-/merge_requests/2179/diffs?commit_id=a029c46acb157b10870e62ff2d5bb3ea2822b34e

So as @DouweM alluded, I think the fix is just dropping that `run_async(invoker.prepare(session))` statement, and double-checking that `meltano config <plugin> test` which actually relies on that `invoker` still works as expected.

Resolving patch:
diff --git a/src/meltano/cli/config.py b/src/meltano/cli/config.py
--- a/src/meltano/cli/config.py
+++ b/src/meltano/cli/config.py
@@ -102,7 +102,6 @@ def config(  # noqa: WPS231
                 plugins_service=plugins_service,
             )
             invoker = PluginInvoker(project, plugin)
-            run_async(invoker.prepare(session))
         else:
             settings = ProjectSettingsService(
                 project, config_service=plugins_service.config_service
@@ -320,8 +319,8 @@ def test(ctx):
     session = ctx.obj["session"]
 
     async def _validate():  # noqa: WPS430
-        async with invoker.prepared(session):
-            plugin_test_service = PluginTestServiceFactory(invoker).get_test_service()
+        plugin_test_service = PluginTestServiceFactory(invoker).get_test_service()
+        async with plugin_test_service.plugin_invoker.prepared(session):
             return await plugin_test_service.validate()
 
     try:


Test patch: