Repository: meltano/meltano

Issue:
`meltano lock` -> variant 'original' is not known to Meltano
I'm getting the following error when I'm trying to run a `meltano lock --all` on [the squared project](https://github.com/meltano/squared/blob/4d0151cf714d9158617d6d76757d21a29c3a38f7/data/extract/extractors.meltano.yml#L63) (note, I did first manually updated to remove executable and namespace which were defined in my project). When I commented out the `inherit_from` children it works, the plugin is locked, then uncommenting the children again makes it fail again so I'm pretty sure its related to the inheriting.

```
File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 521, in get_parent
    parent, source = self.find_parent(plugin)
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 510, in find_parent
    raise error
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 505, in find_parent
    return (self._get_parent_from_hub(plugin), DefinitionSource.HUB)
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/project_plugins_service.py", line 441, in _get_parent_from_hub
    return self.hub_service.get_base_plugin(plugin, variant_name=plugin.variant)
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/plugin_discovery_service.py", line 136, in get_base_plugin
    plugin = project_plugin.custom_definition or self.find_definition(
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/core/hub/client.py", line 189, in find_definition
    raise HubPluginVariantNotFound(
meltano.core.hub.client.HubPluginVariantNotFound: Extractor 'tap-github' variant 'original' is not known to Meltano. Variants: ['singer-io (default)', 'meltanolabs']

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/pnadolny/.virtualenvs/meltano/lib/python3.8/site-packages/meltano/cli/__init__.py", line 74, in _run_cli
    raise CliError(str(err)) from err
meltano.cli.utils.CliError: Extractor 'tap-github' variant 'original' is not known to Meltano. Variants: ['singer-io (default)', 'meltanolabs']
Extractor 'tap-github' variant 'original' is not known to Meltano. Variants: ['singer-io (default)', 'meltanolabs']
```


Discussion:
Additionally I'm getting `Orchestrator 'airflow' variant 'original' is not known to Meltano. Variants: ['apache (default)']` from a default installation of Airflow. I know that its hard because when I originally installed it there wasnt a variant name for airflow but now there is. Is there a way for us to resolve that its the default variant? Maybe that wont work because the default could change, maybe using the discovery.yml when we think were migrating to >2.0.0?

Or we could log it better and say something like "I notice youre migrating and you dont have a variant, go to the Migration guide and learn how to define a variant" and we write up the steps for locking post migration and adding a variant name:

- remove executable/namespace if defined
- add variant using one available on MeltanoHub. We can list the mapping like airflow -> apache, dbt -> dbt-labs, etc. for plugins that just got variant names
- run `meltano lock --all`
Thanks for logging @pnadolny13!

So the `inherit_from` behavior is a more or less serious bug.

For the other one related to plugins added with the "original" variant pre-2.0, I like your suggestion to improve the error message with instructions and the new default variant names.
@edgarrmondragon ok cool, I didnt know if they were going to end up being related but should I spin the messaging/docs one off as a separate issue?
@pnadolny13 yeah, please do. If only just to keep the respective changesets smaller.
@edgarrmondragon new issue created for the log message/migration guide entry https://github.com/meltano/meltano/issues/6360.
@pnadolny13 Just to confirm, you're saying that `meltano lock --all` worked with only this:

```yaml
plugins:
  extractors:
  - name: tap-github
    pip_url: git+https://github.com/MeltanoLabs/tap-github@d99378778c0cebc446c12b552ee4fd386fdc2610
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: auth_token
      kind: password
```

But not when this was (re)added:

```yaml
plugins:
  extractors:
  # ...
  - name: tap-github-meltano
    inherit_from: tap-github
    config:
      organizations:
      - MeltanoLabs
      - meltano
      stream_maps:
        issues:
          __filter__: record['type'] = 'issue'
```

I wonder why in that first case with only `tap-github`, we didn't hit the same error as https://github.com/meltano/meltano/issues/6360, which I'd expect any time there's a `name` but no `variant` (and no `namespace` to indicate a custom plugin, or `inherit_from` to indicate an explicitly inherited one).

> remove executable/namespace if defined

I don't think we should recommend this: `namespace` is still supported and valuable, as it indicates custom plugins that shouldn't be looked up on the Hub (see also https://github.com/meltano/meltano/issues/3296#issuecomment-1166157626 about why we can't just remove that property).
@DouweM The second one is failing because the plugin is inherited, and thus not requiring a variant. In this case, the locking should altogether be skipped according to the lockfile spec, i.e. not adding lockfiles for inherited plugins, only the parent, so that's the bug i.m.o.: `meltano lock` should skip inherited plugins.

To be more clear: my naive implementation of `meltano lock` first _tried to_ pull all plugin definitions from the Hub to make sure the `ProjectPlugin.parent` instance is actually the latest definition when passed to the lockfile manager. I think I got a quick fix for that.
@edgarrmondragon thanks for hopping on this - assigning to you for the current iteration.

Just to be clear, the scope of this is to:

* update `meltano lock` to skip inherited plugins
* update https://docs.meltano.com/concepts/plugins#lock-artifacts to indicate as much

That seem accurate?
> Just to be clear, the scope of this is to:
> 
> * update `meltano lock` to skip inherited plugins
> * update https://docs.meltano.com/concepts/plugins#lock-artifacts to indicate as much
> 
> That seem accurate?

@tayloramurphy yup, that is right


Resolving patch:
diff --git a/src/meltano/cli/lock.py b/src/meltano/cli/lock.py
--- a/src/meltano/cli/lock.py
+++ b/src/meltano/cli/lock.py
@@ -64,14 +64,12 @@ def lock(
         tracker.track_command_event(CliEvent.aborted)
         raise CliError("Exactly one of --all or plugin name must be specified.")
 
-    with plugins_service.use_preferred_source(DefinitionSource.HUB):
-        try:
-            # Make it a list so source preference is not lazily evaluated.
-            plugins = list(plugins_service.plugins())
-
-        except Exception:
-            tracker.track_command_event(CliEvent.aborted)
-            raise
+    try:
+        # Make it a list so source preference is not lazily evaluated.
+        plugins = list(plugins_service.plugins())
+    except Exception:
+        tracker.track_command_event(CliEvent.aborted)
+        raise
 
     if plugin_name:
         plugins = [plugin for plugin in plugins if plugin.name in plugin_name]
@@ -86,7 +84,14 @@ def lock(
         descriptor = f"{plugin.type.descriptor} {plugin.name}"
         if plugin.is_custom():
             click.secho(f"{descriptor.capitalize()} is a custom plugin", fg="yellow")
+        elif plugin.inherit_from is not None:
+            click.secho(
+                f"{descriptor.capitalize()} is an inherited plugin", fg="yellow"
+            )
         else:
+            plugin.parent = None
+            with plugins_service.use_preferred_source(DefinitionSource.HUB):
+                plugin = plugins_service.ensure_parent(plugin)
             try:
                 lock_service.save(plugin, exists_ok=update)
             except LockfileAlreadyExistsError as err:


Test patch:
diff --git a/tests/meltano/cli/test_lock.py b/tests/meltano/cli/test_lock.py
--- a/tests/meltano/cli/test_lock.py
+++ b/tests/meltano/cli/test_lock.py
@@ -93,12 +93,12 @@ def test_lockfile_update_extractors(
         cli_runner: CliRunner,
         project: Project,
         tap: ProjectPlugin,
-        alternative_tap: ProjectPlugin,
+        inherited_tap: ProjectPlugin,
         hub_endpoints: MeltanoHubService,
     ):
         lockfiles = list(project.root_plugins_dir().glob("./*/*.lock"))
-        # 2 taps, 1 target
-        assert len(lockfiles) == 3
+        # 1 tap, 1 target
+        assert len(lockfiles) == 2
 
         result = cli_runner.invoke(
             cli,
@@ -107,4 +107,4 @@ def test_lockfile_update_extractors(
         assert result.exit_code == 0
         assert "Lockfile exists" not in result.output
         assert "Locked definition for extractor tap-mock" in result.output
-        assert "Locked definition for extractor tap-mock--singer-io" in result.output
+        assert "Extractor tap-mock-inherited is an inherited plugin" in result.output