Repository: meltano/meltano

Issue:
Telemetry - Invoke services dont have plugin context or structured event
When running a service plugin like superset that blocks for a long time and is then exited using CTRL+C I'm not able to track the plugin context. I think whats happening is:

- `meltano invoke superset:ui` (blocking)
- CLI started event fired when started up. Plugin context isnt available yet
- build dashboards
- CTRL+C
- <MISSING CLI completed/aborted/etc. with plugin context>
- <MISSING structured legacy event>

I think the fact that I need to CTRL+C to close the blocking service plugin means the structured event isnt fire because its not a successful exit and with the new unstructured events we dont get a plugin context. I dont get enough info for these events to be useful, all I know its that `invoke` ran but nothing about what plugin/command (i.e. `ui`)/etc. This is likely undercounting usage of Airflow also because we never get to the successful firing of structured events either. 

Would it be possible to include the plugin context on the first event?

cc @pandemicsyn @tayloramurphy @WillDaSilva 


Discussion:
@pandemicsyn @tayloramurphy @pnadolny13 Is there ever a situation where we want to remove context? Currently we use `Tracker.add_context` and `Tracker.with_context` to permanently and temporarily add context respectively.

I ask because if we never want to remove context (i.e. only ever permanently add it) then these sorts of things can be handled with #5986. The first time any context (e.g. the plugin context) is created it would be added permanently to the list of contexts that will be sent on any future event. Then if the process exits for almost any reason we'll get those contexts.
@pnadolny13 two options.

1. I can move the cli "started" event to fire later *after* we have plugin context , that means theres a small chance that if you run `meltano invoke` and immediately trigger an exception some how , that we might miss it. 
2. We can evolve the schema and immediately send a "intialized" event, followed by a "started" event once we have the plugins. 

I think 1 makes the most sense right now (so that we can reserve doing a cli initialized as part of https://github.com/meltano/meltano/issues/6041 or #5986 ) 
> @pandemicsyn @tayloramurphy @pnadolny13 Is there ever a situation where we want to remove context? Currently we use Tracker.add_context and Tracker.with_context to permanently and temporarily add context respectively.

I can't think of any cases off the top of my head where we'd need to unbind a context. As long as we stick our pattern of binding PluginContexts to the events/things actually using them. i.e. in `meltano run` plugin context is related to the block events, in `etl` its related to the cli event.
>Is there ever a situation where we want to remove context?

@WillDaSilva I'd defer to florian on that but I'd say probably not.

>I think 1 makes the most sense right now.

@pandemicsyn yeah option 1 seems fine to me.

For more context `meltano run` sends a block initialized event with plugin context but doesnt send an exit event at the block or cli level. I think we're telling people to run services like airflow/superset with invoke anyways but wanted to add more context.

@pnadolny13 ok hotfix in bound.

Resolving patch:
diff --git a/src/meltano/cli/invoke.py b/src/meltano/cli/invoke.py
--- a/src/meltano/cli/invoke.py
+++ b/src/meltano/cli/invoke.py
@@ -78,17 +78,18 @@ def invoke(
     \b\nRead more at https://docs.meltano.com/reference/command-line-interface#invoke
     """
     tracker = Tracker(project)
-    cmd_ctx = cli_context_builder(
-        "invoke",
-        None,
-        plugin_type=plugin_type,
-        dump=dump,
-        list_commands=list_commands,
-        containers=containers,
-        print_var=print_var,
+    # the `started` event is delayed until we've had a chance to try to resolve the requested plugin
+    tracker.add_contexts(
+        cli_context_builder(
+            "invoke",
+            None,
+            plugin_type=plugin_type,
+            dump=dump,
+            list_commands=list_commands,
+            containers=containers,
+            print_var=print_var,
+        )
     )
-    with tracker.with_contexts(cmd_ctx):
-        tracker.track_command_event(cli_tracking.STARTED)
 
     try:
         plugin_name, command_name = plugin_name.split(":")
@@ -105,13 +106,17 @@ def invoke(
         plugin = plugins_service.find_plugin(
             plugin_name, plugin_type=plugin_type, invokable=True
         )
-    except PluginNotFoundError as err:
-        with tracker.with_contexts(cmd_ctx):
-            tracker.track_command_event(cli_tracking.ABORTED)
-        raise err
+        tracker.add_contexts(PluginsTrackingContext([(plugin, command_name)]))
+        tracker.track_command_event(cli_tracking.STARTED)
+    except PluginNotFoundError:
+        # if the plugin is not found, we fire started and aborted tracking events together to keep tracking consistent
+        tracker.track_command_event(cli_tracking.STARTED)
+        tracker.track_command_event(cli_tracking.ABORTED)
+        raise
 
     if list_commands:
         do_list_commands(plugin)
+        tracker.track_command_event(cli_tracking.COMPLETED)
         return
 
     invoker = invoker_factory(project, plugin, plugins_service=plugins_service)
@@ -130,17 +135,13 @@ def invoke(
             )
         )
     except Exception as invoke_err:
-        with tracker.with_contexts(cmd_ctx):
-            tracker.track_command_event(cli_tracking.FAILED)
+        tracker.track_command_event(cli_tracking.FAILED)
         raise invoke_err
 
-    with tracker.with_contexts(
-        cmd_ctx, PluginsTrackingContext([(plugin, command_name)])
-    ):
-        if exit_code == 0:
-            tracker.track_command_event(cli_tracking.COMPLETED)
-        else:
-            tracker.track_command_event(cli_tracking.FAILED)
+    if exit_code == 0:
+        tracker.track_command_event(cli_tracking.COMPLETED)
+    else:
+        tracker.track_command_event(cli_tracking.FAILED)
     sys.exit(exit_code)
 
 


Test patch: