Repository: meltano/meltano

Issue:
Disallow calling a `mapper` by name in `meltano run` and alert to use a `mapping` name
The document misses a step for mapper configuration. Property `mapping_name` has to be added to the mappers extras, and its value need to match with the name under mappings. It doesn't seem been documented anywhere.

```yaml
plugins:
  mappers:
  - name: transform-field
    variant: transferwise
    pip_url: pipelinewise-transform-field
    executable: transform-field
    mapping_name: transform-field-hash
    mappings:
    - name: transform-field-hash
      config:
        transformations:
          - field_id: Name
            tap_stream_name: dbo-Test
            type: MASK-HIDDEN
```


Discussion:
@edgarrmondragon can you confirm this is accurate? In my demo project I didn't have to add `mapping_name` to the plugin definition. Using `meltano run <tap> transform-field-hash <target>` worked fine when I had a config like:

```
plugins:
mappers:
  - name: transform-field
    variant: transferwise
    pip_url: pipelinewise-transform-field
    mappings:
    - name: hash-url-data
      config:
        transformations:
        - field_id: html_url
          tap_stream_name: commits
          type: HASH
```

@lidong-zhao where are you finding this requirement?
@tayloramurphy from here: https://docs.meltano.com/concepts/plugins#mappers
@lidong-zhao so the text in that link is:

> Mappers are unique in that after install you don’t invoke them directly. Instead you define mappings by name and add a config object for each mapping. This config object is passed to the mapper when the **mapping name** is called as part of a [meltano run](https://docs.meltano.com/reference/command-line-interface#run) invocation. Note that this differs from other plugins, as you’re not invoking a plugin name - but referencing the mapping name instead. Additionally, the requirements for the config object itself will vary by plugin.

which is talking about the key `name` under `mappings`. 

Do you have an example or stack trace where the run is failing because of a missing key called `mapping_name`?
2022-06-14T17:15:35.460549Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.464533Z [info     ]   File "C:\Python38\lib\runpy.py", line 194, in _run_module_as_main cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.468532Z [info     ]     return _run_code(code, main_globals, None, cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.474547Z [info     ]   File "C:\Python38\lib\runpy.py", line 87, in _run_code cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.479533Z [info     ]     exec(code, run_globals)    cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.484533Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\Scripts\transform-field.exe\__main__.py", line 7, in <module> cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.490534Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\lib\site-packages\transform_field\__init__.py", line 334, in main cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.495532Z [info     ]     raise exc                  cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.500532Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\lib\site-packages\transform_field\__init__.py", line 327, in main cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.507533Z [info     ]     main_impl()                cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.512532Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\lib\site-packages\transform_field\__init__.py", line 310, in main_impl cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.517533Z [info     ]     args = utils.parse_args(REQUIRED_CONFIG_KEYS) cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.523532Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\lib\site-packages\transform_field\utils.py", line 54, in parse_args cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.528531Z [info     ]     check_config(args.config, required_config_keys) cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.533532Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\lib\site-packages\singer\utils.py", line 188, in check_config cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.538544Z [info     ]     missing_keys = [key for key in required_keys if key not in config] cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.543531Z [info     ]   File "C:\Repos\meltano\src\lidong\.meltano\mappers\transform-field\venv\lib\site-packages\singer\utils.py", line 188, in <listcomp> cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.547531Z [info     ]     missing_keys = [key for key in required_keys if key not in config] cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.551533Z [info     ] TypeError: argument of type 'NoneType' is not iterable cmd_type=elb consumer=True name=transform-field producer=True stdio=stderr string_id=transform-field
2022-06-14T17:15:35.573533Z [info     ] INFO Columns Fetched           cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql
2022-06-14T17:15:35.578534Z [info     ] INFO Catalog ready             cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stderr string_id=tap-mssql
2022-06-14T17:15:37.738192Z [warning  ] Intermediate block in sequence failed.
2022-06-14T17:15:37.758572Z [info     ] {"type": "STATE", "value": {"currently_syncing": null}} cmd_type=elb consumer=False name=tap-mssql producer=True stdio=stdout string_id=tap-mssql
2022-06-14T17:15:37.764572Z [debug    ] Deleted configuration at C:\Repos\meltano\src\lidong\.meltano\run\tap-mssql\tap.12ab724a-caa7-4677-9db0-990faf8f2f9a.config.json
2022-06-14T17:15:37.766573Z [debug    ] Deleted configuration at C:\Repos\meltano\src\lidong\.meltano\run\transform-field\mapper.dd4d1257-a575-4651-b5b8-95344992261c.config.json
2022-06-14T17:15:37.767570Z [info     ] time=2022-06-14 10:15:36 name=target_snowflake level=INFO message=Getting catalog objects from table cache... cmd_type=elb consumer=True name=target-snowflake producer=False stdio=stderr string_id=target-snowflake
2022-06-14T17:15:37.776572Z [debug    ] Deleted configuration at C:\Repos\meltano\src\lidong\.meltano\run\target-snowflake\target.34ed7931-20a2-44f9-87df-d229ffc53cea.config.json
2022-06-14T17:15:37.798011Z [error    ] Unexpected completion sequence in ExtractLoadBlock set. Intermediate block (likely a mapper) failed. 
without mapping_name property, it will generate empty config file, I figured that out based on the code in mapper.py:
```
    @staticmethod
    def _get_mapping_config(extra_config: dict) -> Optional[dict]:
        for mapping in extra_config.get("_mappings", []):
            if mapping.get("name") == extra_config.get("_mapping_name"):
                return mapping["config"]
```
@lidong-zhao thanks for that code snippet. Comes from https://github.com/meltano/meltano/blob/main/src/meltano/core/plugin/singer/mapper.py#L64 I _think_ it's possible that's a red herring because that key is generated internally from what I can tell in this file https://github.com/meltano/meltano/blob/main/src/meltano/core/meltano_file.py#L131

Can you share the command you were running and more of your `meltano.yml` to further debug? 

@edgarrmondragon am I missing anything here 😅 ?
I didn't use command, I directly updated the meltano.yml file. 
@lidong-zhao sorry, I meant the command you ran that generated the stack trace. Were you using `meltano run`?
yes. meltano run tap-upmssql transform-field target-snowflake
@lidong-zhao Ok, great. Thank you for sharing that! 

I've tested locally with 2.0 and a `mapping_name` defined in your meltano.yml file is _not_ used or referenced when you're using `meltano run`.  Based on the yaml in your original comment https://github.com/meltano/meltano/issues/6189#issue-1270988582 the command should be `meltano run tap-upmssql transform-field-hash target-snowflake`. 

`transform-field-hash` is referencing the the `name` key under the `mapping` key. i.e.:

```yaml
    mappings:
    - name: transform-field-hash # meltano run uses this
```

I think you'll find that if you delete `mapping_name` from your meltano.yml and then update your `meltano run` command to use `transform-field-hash` it should work (barring any other configuration errors). Let me know if that works!
It works without mapping_name if I use transform-field-hash. Thanks.
@tayloramurphy it seems that using the mapper's name instead of the mapping's when the mapper is not itself given a configuration leads to this error. Do you think we should document that the mapper itself can be given a `config`?
it's better to validate the input first, in my case, you could throw exception like 'no mapping {name}" found, then it would be easily to figure out the problem.
> @tayloramurphy it seems that using the mapper's name instead of the mapping's when the mapper is not itself given a configuration leads to this error. Do you think we should document that the mapper itself can be given a `config`?

@edgarrmondragon we should document that I suppose, but it's not clear to me how that would interact with `mappings` then. If you're supplying config to the mapper is it overriden or combined with the config for a given mapping? Doesn't running the mapper by calling a specific mapping generate the config for that mapper and pass that to the plugin?
@edgarrmondragon ping on the question above 👆 
> If you're supplying config to the mapper is it overriden or combined with the config for a given mapping?

@tayloramurphy that's a good question. I don't know off the top of my head 😅. I assume they're not merged in any way and only the `mapping` config is used in that case. It's a bit blurry to me how the mapper-mapping relationship works. Regardless, the current state is confusing and we should probably discourage (or fail?) calling a mapper by name in `meltano run`.
@edgarrmondragon I think that makes sense. I'll update the title of this issue and scope it to the following:

* Hard fail if a `mapper` is called
* Point users to call a specific `mapping`

Looking at the [docs](https://docs.meltano.com/concepts/plugins#mappers) it's not clear how we could update it to be more descriptive, but I think we can be explict in the code
> * Hard fail if a `mapper` is called
> * Point users to call a specific `mapping`

👍 I like it!

Resolving patch:
diff --git a/src/meltano/core/block/parser.py b/src/meltano/core/block/parser.py
--- a/src/meltano/core/block/parser.py
+++ b/src/meltano/core/block/parser.py
@@ -290,11 +290,25 @@ def _find_next_elb_set(  # noqa: WPS231, WPS213
                     mapping=self._mappings_ref.get(next_block),
                     idx=next_block,
                 )
-                blocks.append(
-                    builder.make_block(
-                        plugin,
+                # Checks to see if the mapper plugin name is the same as the mappings name
+                # If they both match then a validation error is raised because the
+                # meltano run command needs the mappings name to obtain the settings to
+                # pass to the parent mapper plugin.  We also want to fail if the user names them
+                # the same to stop errors due to ambiguous commands.
+                if plugin.name == self._mappings_ref.get(next_block):
+                    self.log.warning(
+                        "Found unexpected mapper plugin name. ",
+                        plugin_name=plugin.name,
+                    )
+                    raise BlockSetValidationError(
+                        f"Expected unique mappings name not the mapper plugin name: {plugin.name}."
+                    )
+                else:
+                    blocks.append(
+                        builder.make_block(
+                            plugin,
+                        )
                     )
-                )
             elif plugin.type == PluginType.LOADERS:
                 self.log.debug("blocks", offset=offset, idx=next_block)
                 blocks.append(builder.make_block(plugin))


Test patch:
diff --git a/tests/meltano/cli/test_run.py b/tests/meltano/cli/test_run.py
--- a/tests/meltano/cli/test_run.py
+++ b/tests/meltano/cli/test_run.py
@@ -1085,6 +1085,64 @@ def test_run_mapper_config(
             assert result.exit_code == 1
             assert "Error: Block not-a-valid-mapping-name not found" in result.stderr
 
+        # test mapper/mapping name collision detection - mapper plugin name no mappings
+        project_add_service.add(
+            PluginType.MAPPERS, "mapper-collision-01", inherit_from=mapper.name
+        )
+        args = ["run", tap.name, "mapper-collision-01", target.name]
+        with mock.patch.object(SingerTap, "discover_catalog"), mock.patch.object(
+            SingerTap, "apply_catalog_rules"
+        ), mock.patch(
+            "meltano.core.plugin_invoker.asyncio"
+        ) as asyncio_mock2, mock.patch(
+            "meltano.core.block.parser.ProjectPluginsService",
+            return_value=project_plugins_service,
+        ):
+            asyncio_mock2.create_subprocess_exec = create_subprocess_exec
+            with pytest.raises(
+                Exception,
+                match="block violates set requirements: Expected unique mappings name not the mapper plugin name: mapper-collision-01",
+            ):
+                result = cli_runner.invoke(cli, args, catch_exceptions=False)
+                assert result.exit_code == 1
+
+        # test mapper/mapping name collision detection - mappings name same a mapper plugin name
+        project_add_service.add(
+            PluginType.MAPPERS,
+            "mapper-collision-02",
+            inherit_from=mapper.name,
+            mappings=[
+                {
+                    "name": "mapper-collision-02",
+                    "config": {
+                        "transformations": [
+                            {
+                                "field_id": "author_email1",
+                                "tap_stream_name": "commits1",
+                                "type": "MASK-HIDDEN",
+                            }
+                        ]
+                    },
+                }
+            ],
+        )
+        args = ["run", tap.name, "mapper-collision-02", target.name]
+        with mock.patch.object(SingerTap, "discover_catalog"), mock.patch.object(
+            SingerTap, "apply_catalog_rules"
+        ), mock.patch(
+            "meltano.core.plugin_invoker.asyncio"
+        ) as asyncio_mock2, mock.patch(
+            "meltano.core.block.parser.ProjectPluginsService",
+            return_value=project_plugins_service,
+        ):
+            asyncio_mock2.create_subprocess_exec = create_subprocess_exec
+            with pytest.raises(
+                Exception,
+                match="block violates set requirements: Expected unique mappings name not the mapper plugin name: mapper-collision-02",
+            ):
+                result = cli_runner.invoke(cli, args, catch_exceptions=False)
+                assert result.exit_code == 1
+
         # create duplicate mapping name - should also fail
         project_add_service.add(
             PluginType.MAPPERS,