Repository: meltano/meltano

Issue:
Ability to test `meltano add` on a plugin prior to a Hub PR being merged
I'd love to have a way to test `meltano add ...` on a plugin before we publish to the Hub.

Any ideas? I initially thought to use the `hub_url` overrides to target the preview deployment, but I am pretty sure we are not publishing the JSON API to Netlify. We also can't use the S3 files since many of these are run from community forks and won't have any S3 publish permissions.

Perhaps we need to implement a `--from=HUB_YAML_FILE_URL` arg in the CLI for `meltano add` or else make it process to download the yaml file from the hub directly into the location where the lockfile would be?

cc @edgarrmondragon, @pnadolny13 - I think you two are the more knowlegeble on these processes. Any ideas?
feature: developing plugins locally before publishing to hub
### Feature scope

Configuration (settings parsing, validation, etc.)

### Description

**As a user, I want to develop plugins with dependencies before publishing these plugins to the HUB**

### Background

We use a `file` bundle type plugin to distribute analytics:  `analyze-google-analytics`

We use a `file` bundle type plugin to distribute models:  `dbt-google-analytics`

We use an `extractor` type plugin to import data:  `tap-google-analytics`

To work correctly:  `analyze` requires `models` requires `tap`


## problem

The developer wants to create `analyze` and `models` at the same time, using an existing `extractor`

But `meltano add file analyze-google-analytics` cannot be built and tested until `models-google-analytics` is published on the HUB


## proposal

The developer creates the plugin definitions and adds using a local file path

NB - is `--from_ref` undocumented https://github.com/meltano/meltano/issues/3208

`meltano add file models-google-analytics --from_ref=../projects/models-google-analytics.yml`
`meltano add file analyze-google-analytics --from_ref=../projects/analyze-google-analytics.yml`

Regular plugin yml definition
`
/projects/analyze-google-analytics.yml
`

Regular plugin yml definition
`
/projects/plugins/models-google-analytics.yml
`


## Related issues

https://github.com/meltano/meltano/issues/3208

"using local hub instance" is not available for local developers https://github.com/meltano/meltano/issues/5997#issuecomment-1146358235

 


Discussion:
> Perhaps we need to implement a --from=HUB_YAML_FILE_URL arg in the CLI for meltano add or else make it process to download the yaml file from the hub directly into the location where the lockfile would be?

What about merging it directly into the `meltano.yml`, kinda of like how `meltano add --custom` works?
@aaronsteers I could have sworn that the preview hub url worked but youre right, I got something like "utility matatika" not found when I tried for that PR.

>Perhaps we need to implement a --from=HUB_YAML_FILE_URL arg in the CLI for meltano add or else make it process to download the yaml file from the hub directly into the location where the lockfile would be?

....

>What about merging it directly into the meltano.yml, kinda of like how meltano add --custom works?

Yeah I like those ideas of having a way to point to the hub yaml file and let it install from there as a custom plugin and pull out all the relevant keys that it needs to install.
> > Perhaps we need to implement a --from=HUB_YAML_FILE_URL arg in the CLI for meltano add or else make it process to download the yaml file from the hub directly into the location where the lockfile would be?
> 
> What about merging it directly into the `meltano.yml`, kinda of like how `meltano add --custom` works?

This might work... I'm not sure. I considered it, but backed down just because it seemed pretty messy with the amount of time I had on Tuesday night to test. Definitely worth a shot, I think.

There would be a ton of extra properties that meltano might (?) complain about, and manually you'd need to at least indent the code and add under a bullet.
I like the idea. Only the `docs` and `logo_url` would be wrong because the need the base URL, but otherwise all the info returned from the API would be there 👍. It's a low effort too AFAICT.
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the `evergreen` label, or request that it be added.

IMO still relevant and related to #6615.
@aphethean1 it seems like adding `--from_ref` would basically solve this problem, right? We'd talked about that in the original spec but it fell out of scope given the time constraints. 

I don't think we'll prioritize this ourselves right now, but I'm in favor of it! cc @aaronsteers @edgarrmondragon 
@tayloramurphy @aphethean1 Note that

> "using local hub instance" is not available for local developers https://github.com/meltano/meltano/issues/5997#issuecomment-1146358235

should no longer the case and developing file bundles with a local Hub instance should be working now: https://github.com/meltano/meltano/pull/5999#issuecomment-1146404199
@tayloramurphy yes, I think `--from_ref` would be ideal.  Running the hub locally for this development use case seems pretty heavy!
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the `evergreen` label, or request that it be added.

Still relevant.

Side note: If we end up going with `--from_ref` to resolve this issue, it should actually be `--from-ref`. Relates to:
- https://github.com/meltano/meltano/issues/2253

Resolving patch:
diff --git a/src/meltano/cli/add.py b/src/meltano/cli/add.py
--- a/src/meltano/cli/add.py
+++ b/src/meltano/cli/add.py
@@ -3,8 +3,11 @@
 from __future__ import annotations
 
 import typing as t
+from pathlib import Path
+from urllib.parse import urlparse
 
 import click
+import requests
 
 from meltano.cli.params import pass_project
 from meltano.cli.utils import (
@@ -19,6 +22,7 @@
 from meltano.core.plugin_install_service import PluginInstallReason
 from meltano.core.project_add_service import ProjectAddService
 from meltano.core.tracking.contexts import CliEvent, PluginsTrackingContext
+from meltano.core.yaml import yaml
 
 if t.TYPE_CHECKING:
     from meltano.core.plugin.project_plugin import ProjectPlugin
@@ -26,6 +30,25 @@
     from meltano.core.tracking import Tracker
 
 
+def _load_yaml_from_ref(_ctx, _param, value: str | None) -> dict:
+    if not value:
+        return
+
+    try:
+        url = urlparse(value)
+        if url.scheme and url.netloc:
+            response = requests.get(value, timeout=10)
+            response.raise_for_status()
+            content = response.text
+        else:
+            content = Path(value).read_text()
+
+    except (ValueError, FileNotFoundError, IsADirectoryError) as e:
+        raise click.BadParameter(e) from e
+
+    return yaml.load(content) or {}
+
+
 @click.command(  # noqa: WPS238
     cls=PartialInstrumentedCmd,
     short_help="Add a plugin to your project.",
@@ -52,6 +75,12 @@
         "Usage:\b\n\nadd <type> <inherit-from> --as <name>"
     ),
 )
+@click.option(
+    "--from-ref",
+    "plugin_yaml",
+    callback=_load_yaml_from_ref,
+    help="Reference a plugin defintion to add from.",
+)
 @click.option(
     "--python",
     help=(
@@ -83,6 +112,7 @@ def add(  # noqa: WPS238
     inherit_from: str | None = None,
     variant: str | None = None,
     as_name: str | None = None,
+    plugin_yaml: dict | None = None,
     python: str | None = None,
     **flags,
 ):
@@ -135,6 +165,7 @@ def add(  # noqa: WPS238
                     variant=variant,
                     custom=flags["custom"],
                     add_service=add_service,
+                    plugin_yaml=plugin_yaml,
                 ),
             )
         except Exception:
diff --git a/src/meltano/cli/utils.py b/src/meltano/cli/utils.py
--- a/src/meltano/cli/utils.py
+++ b/src/meltano/cli/utils.py
@@ -15,8 +15,8 @@
 
 from meltano.core.error import MeltanoConfigurationError
 from meltano.core.logging import setup_logging
-from meltano.core.plugin import PluginType
-from meltano.core.plugin.error import PluginNotFoundError
+from meltano.core.plugin.base import PluginDefinition, PluginType
+from meltano.core.plugin.error import InvalidPluginDefinitionError, PluginNotFoundError
 from meltano.core.plugin_install_service import (
     PluginInstallReason,
     PluginInstallService,
@@ -253,7 +253,7 @@ def _prompt_plugin_settings(plugin_type):
     return settings
 
 
-def add_plugin(
+def add_plugin(  # noqa: C901
     plugin_type: PluginType,
     plugin_name: str,
     *,
@@ -263,6 +263,7 @@ def add_plugin(
     inherit_from=None,
     custom=False,
     lock=True,
+    plugin_yaml=None,
 ):
     """Add Plugin to given Project."""
     if custom:
@@ -279,6 +280,25 @@ def add_plugin(
     else:
         plugin_attrs = {}
 
+    if plugin_yaml is not None:
+        try:
+            plugin_definition = PluginDefinition.parse(
+                {
+                    "plugin_type": plugin_type,
+                    **plugin_yaml,
+                },
+            )
+        except TypeError as e:
+            raise InvalidPluginDefinitionError(plugin_yaml) from e
+
+        # exclude unspecified properties
+        plugin_definition.extras.clear()
+
+        plugin_attrs = plugin_definition.canonical()
+
+        plugin_name = plugin_attrs.pop("name")
+        variant = plugin_attrs.pop("variant")
+
     try:
         plugin = add_service.add(
             plugin_type,
diff --git a/src/meltano/core/plugin/error.py b/src/meltano/core/plugin/error.py
--- a/src/meltano/core/plugin/error.py
+++ b/src/meltano/core/plugin/error.py
@@ -1,5 +1,9 @@
 from __future__ import annotations
 
+import inspect
+
+from meltano.core.plugin.base import PluginDefinition
+
 from . import PluginRef
 
 
@@ -56,3 +60,27 @@ class PluginExecutionError(Exception):
 
 class PluginLacksCapabilityError(Exception):
     """Base exception when a plugin lacks a requested capability."""
+
+
+class InvalidPluginDefinitionError(Exception):
+    """Base exception when a plugin definition is invalid."""
+
+    def __init__(self, definition=None):
+        """Construct a new "InvalidPluginDefinitionError instance.
+
+        Args:
+            definition: Plugin definition.
+        """
+        fullargspec = inspect.getfullargspec(PluginDefinition.__init__)  # noqa: WPS609
+        _, _plugin_type, *required_properties = fullargspec.args
+
+        if isinstance(definition, dict):
+            missing_properties = ", ".join(
+                [p for p in required_properties if p not in definition],
+            )
+            self.reason = f"missing properties ({missing_properties})"
+        else:
+            self.reason = "incorrect format"
+
+    def __str__(self):
+        return f"Plugin definition is invalid: {self.reason}"


Test patch:
diff --git a/tests/fixtures/cli.py b/tests/fixtures/cli.py
--- a/tests/fixtures/cli.py
+++ b/tests/fixtures/cli.py
@@ -19,6 +19,7 @@
 
 
 current_dir = Path(__file__).parent
+plugins_dir = current_dir / "plugins"
 
 
 class MeltanoCliRunner(CliRunner):
diff --git a/tests/fixtures/plugins/extractors/tap-custom/test.yml b/tests/fixtures/plugins/extractors/tap-custom/test.yml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/plugins/extractors/tap-custom/test.yml
@@ -0,0 +1,31 @@
+capabilities:
+- catalog
+- discover
+- state
+- about
+- stream-maps
+description: Custom extractor for testing
+domain_url: https://meltano.com/
+keywords:
+- custom
+- test
+label: Custom
+logo_url: /assets/logos/extractors/custom.png
+maintenance_status: development
+name: tap-custom
+namespace: tap_custom
+pip_url: git+https://github.com/MeltanoLabs/tap-custom.git
+quality: gold
+repo: https://github.com/MeltanoLabs/tap-custom
+settings:
+- description: Username
+  label: Username
+  name: username
+- description: Password
+  label: Password
+  name: password
+  sensitive: true
+settings_group_validation:
+- - username
+  - password
+variant: test
diff --git a/tests/meltano/cli/test_add.py b/tests/meltano/cli/test_add.py
--- a/tests/meltano/cli/test_add.py
+++ b/tests/meltano/cli/test_add.py
@@ -10,10 +10,11 @@
 import yaml
 
 from asserts import assert_cli_runner
+from fixtures.cli import plugins_dir
 from meltano.cli import cli
 from meltano.cli.utils import CliError
 from meltano.core.plugin import PluginRef, PluginType, Variant
-from meltano.core.plugin.error import PluginNotFoundError
+from meltano.core.plugin.error import InvalidPluginDefinitionError, PluginNotFoundError
 from meltano.core.plugin.project_plugin import ProjectPlugin
 from meltano.core.plugin_install_service import PluginInstallReason
 from meltano.core.project import Project
@@ -22,6 +23,12 @@
 if t.TYPE_CHECKING:
     from click.testing import CliRunner
 
+plugin_ref = plugins_dir / "extractors" / "tap-custom" / "test.yml"
+fails_on_windows = pytest.mark.xfail(
+    platform.system() == "Windows",
+    reason="Fails on Windows: https://github.com/meltano/meltano/issues/3444",
+)
+
 
 class TestCliAdd:
     @pytest.fixture()
@@ -33,6 +40,11 @@ def reset_project_context(
         shutil.rmtree(".", ignore_errors=True)
         project_init_service.create_files(project)
 
+        project.refresh()
+
+        for plugin_type in PluginType:
+            project.meltano.plugins[plugin_type].clear()
+
     @pytest.mark.order(0)
     @pytest.mark.parametrize(
         ("plugin_type", "plugin_name", "default_variant", "required_plugin_refs"),
@@ -196,16 +208,13 @@ def test_add_transform(self, project: Project, cli_runner):
             "schema": "{{ env_var('DBT_SOURCE_SCHEMA', 'tap_google_analytics') }}",
         }
 
+    @fails_on_windows
     def test_add_files_with_updates(
         self,
         project: Project,
         cli_runner,
         plugin_settings_service_factory,
     ):
-        if platform.system() == "Windows":
-            pytest.xfail(
-                "Fails on Windows: https://github.com/meltano/meltano/issues/3444",
-            )
         # if plugin is locked, we actually wouldn't expect it to update.
         # So we must remove lockfile
         shutil.rmtree("plugins/files", ignore_errors=True)
@@ -252,11 +261,8 @@ def test_add_files_without_updates(self, project: Project, cli_runner):
         # File does not have "managed" header
         assert "This file is managed" not in file_path.read_text()
 
+    @fails_on_windows
     def test_add_files_that_already_exists(self, project: Project, cli_runner):
-        if platform.system() == "Windows":
-            pytest.xfail(
-                "Fails on Windows: https://github.com/meltano/meltano/issues/3444",
-            )
         # dbt lockfile was created in an upstream test. Need to remove.
         shutil.rmtree(project.root_dir("plugins/files"), ignore_errors=True)
         project.root_dir("transform/dbt_project.yml").write_text("Exists!")
@@ -426,6 +432,7 @@ def test_add_inherited(
                 "Extractor 'tap-bar' is not known to Meltano"
             ) in str(res.exception)
 
+    @pytest.mark.usefixtures("reset_project_context")
     def test_add_custom(self, project: Project, cli_runner):
         pip_url = "-e path/to/tap-custom"
         executable = "tap-custom-bin"
@@ -576,7 +583,6 @@ def test_add_no_install(
         project: Project,
         cli_runner,
     ):
-        project.refresh()
         # ensure the plugin is not present
         with pytest.raises(PluginNotFoundError):
             project.plugins.find_plugin(plugin_name, plugin_type=plugin_type)
@@ -635,6 +641,140 @@ def test_add_no_install(
 
                 install_plugin_mock.assert_not_called()
 
+    @pytest.mark.parametrize(
+        "ref",
+        (
+            plugin_ref,
+            (
+                "https://raw.githubusercontent.com/meltano/hub/main/_data/meltano/"
+                f"{plugin_ref.relative_to(plugins_dir)}"
+            ),
+        ),
+        ids=(
+            "local",
+            "remote",
+        ),
+    )
+    @pytest.mark.usefixtures("reset_project_context")
+    @mock.patch("meltano.cli.add.install_plugins")
+    @mock.patch("meltano.cli.add.requests.get")
+    def test_add_from_ref(
+        self,
+        ref_request_mock,
+        install_plugin_mock,
+        ref,
+        project,
+        cli_runner,
+    ):
+        ref_request_mock.return_value.status_code = 200
+        ref_request_mock.return_value.text = plugin_ref.read_text()
+
+        install_plugin_mock.return_value = True
+
+        plugin_name = plugin_ref.parent.name
+        plugin_type = PluginType(plugin_ref.parents[1].name)
+
+        res = cli_runner.invoke(
+            cli,
+            ["add", plugin_type.singular, plugin_name, "--from-ref", ref],
+        )
+        assert_cli_runner(res)
+
+        assert f"Added {plugin_type.singular} '{plugin_name}'" in res.output
+
+        plugin = project.plugins.find_plugin(plugin_name, plugin_type)
+        assert plugin.name == plugin_name
+        assert plugin.variant == "test"
+
+    @pytest.mark.usefixtures("reset_project_context")
+    @pytest.mark.parametrize(
+        (
+            "ref",
+            "invalid_reason",
+        ),
+        (
+            (
+                plugin_ref.name,
+                "No such file or directory: '{ref}'",
+            ),
+            pytest.param(
+                plugin_ref.parent,
+                "Is a directory: '{ref}'",
+                marks=fails_on_windows,
+            ),
+            (
+                "https://:",
+                "Invalid URL '{ref}'",
+            ),
+        ),
+        ids=(
+            "does not exist",
+            "is directory",
+            "invalid url",
+        ),
+    )
+    def test_add_from_ref_invalid_ref(
+        self,
+        ref,
+        invalid_reason,
+        cli_runner,
+    ):
+        res = cli_runner.invoke(
+            cli,
+            ["add", "extractor", "tap-custom", "--from-ref", ref],
+        )
+
+        assert res.exit_code == 2
+        assert invalid_reason.format(ref=ref) in res.stderr
+
+    @pytest.mark.parametrize(
+        (
+            "definition",
+            "invalid_reason",
+        ),
+        (
+            (
+                "test",
+                "incorrect format",
+            ),
+            (
+                {},
+                "missing properties (name, namespace)",
+            ),
+            (
+                {"test-key": "test-value"},
+                "missing properties (name, namespace)",
+            ),
+            (
+                {"name": "tap-custom"},
+                "missing properties (namespace)",
+            ),
+        ),
+        ids=(
+            "incorrect format",
+            "empty",
+            "no required properties",
+            "some required properties",
+        ),
+    )
+    def test_add_from_ref_invalid_definiton(
+        self,
+        definition,
+        invalid_reason,
+        cli_runner,
+    ):
+        with open("test.yml", "w") as f:
+            yaml.dump(definition, f)
+
+        res = cli_runner.invoke(
+            cli,
+            ["add", "extractor", "tap-custom", "--from-ref", f.name],
+        )
+
+        assert res.exit_code == 1
+        assert isinstance(res.exception, InvalidPluginDefinitionError)
+        assert res.exception.reason == invalid_reason
+
     def test_add_with_python_version(self, cli_runner: CliRunner):
         with mock.patch(
             "meltano.core.venv_service.VirtualEnv._resolve_python_path",