Repository: ansible/molecule

Issue:
Can't get my role to run locally anymore (OSError: [Errno 22] Invalid argument)
# Issue Type

- Bug report

# Molecule and Ansible details

```
$ ansible --version && molecule --version
ansible [core 2.12.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/Users/jgeerling/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jgeerling/Library/Python/3.8/lib/python/site-packages/ansible
  ansible collection location = /Users/jgeerling/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/jgeerling/Library/Python/3.8/bin/ansible
  python version = 3.8.9 (default, Feb 18 2022, 07:45:33) [Clang 13.1.6 (clang-1316.0.21.2)]
  jinja version = 3.0.3
  libyaml = False
molecule 3.6.1 using python 3.9 
    ansible:2.12.3
    delegated:3.6.1 from molecule
    docker:1.1.0 from molecule_docker requiring collections: community.docker>=1.9.1
```

Molecule installation method (one of):

- pip

Ansible installation method (one of):

- pip

Detail any linters or test runners used:

N/A, run locally on macOS 12.3.1

# Desired Behavior

If I run `molecule test` inside one of my Ansible roles, I can run `molecule test` successfully.

# Actual Behaviour

```
$ git clone https://github.com/geerlingguy/ansible-role-docker.git
$ molecule test
$ molecule test
INFO     default scenario test matrix: dependency, lint, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO     Performing prerun...
INFO     Set ANSIBLE_LIBRARY=/Users/jgeerling/.cache/ansible-compat/320470/modules:/Users/jgeerling/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/Users/jgeerling/.cache/ansible-compat/320470/collections:/Users/jgeerling/.ansible/collections:/usr/share/ansible/collections
INFO     Set ANSIBLE_ROLES_PATH=/Users/jgeerling/.cache/ansible-compat/320470/roles:/Users/jgeerling/Dropbox/VMs/roles
Traceback (most recent call last):
  File "/opt/homebrew/bin/molecule", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/molecule/command/test.py", line 161, in test
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/opt/homebrew/lib/python3.9/site-packages/molecule/command/base.py", line 111, in execute_cmdline_scenarios
    scenario.config.runtime.prepare_environment(install_local=True)
  File "/opt/homebrew/lib/python3.9/site-packages/ansible_compat/runtime.py", line 334, in prepare_environment
    self._install_galaxy_role(self.project_dir, ignore_errors=True)
  File "/opt/homebrew/lib/python3.9/site-packages/ansible_compat/runtime.py", line 494, in _install_galaxy_role
    if not exists or os.readlink(link_path) != str(target):
OSError: [Errno 22] Invalid argument: '/Users/jgeerling/Dropbox/VMs/roles/geerlingguy.docker'
```

I have my global Ansible roles directory symlinked to `/Users/jgeerling/Dropbox/VMs/roles` for convenience:

```
$ ls -lah /etc/ansible/roles
lrwxr-xr-x  1 root  wheel    34B Mar 22 15:01 /etc/ansible/roles -> /Users/jgeerling/Dropbox/VMs/roles
```

And in my case, I have my roles checked out to directories matching the galaxy names (e.g. `geerlingguy.docker` insteaad of `ansible-role-docker`), but that doesn't seem to matter, as the same failure occurs if I check out the role into any directory on my system.


Discussion:
Testing other versions:

  - Molecule 3.0.8 ✅
  - Molecule 3.2.0 ✅
  - Molecule 3.4.0 ✅
  - Molecule 3.5.2 ✅
  - Molecule 3.6.0 ✅
  - Molecule 3.6.1 ❌
Just realized my `ansible` executable was somehow installed in my system Python (under user/library), so I uninstalled and reinstalled in the correct Python 3.9 path.

Still not able to get Molecule 3.6.1 to work (same error).
Hi @geerlingguy 

I am glad to help you.

Would you mind giving me more detailed information?

I cannot reproduce this issue.

Here is what I tried.

```shell
❯ python -V
Python 3.9.7

❯ pip freeze | grep molecule
molecule==3.6.1
molecule-docker==1.1.0

❯ ansible-galaxy collection list

# /Users/jackzhang/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.docker  2.3.0
community.general 4.7.0

❯ ls -lah /etc/ansible/roles
lrwxr-xr-x  1 root  wheel    30B Apr  6 10:53 /etc/ansible/roles -> /Users/jackzhang/src/tmp/roles
```

Run test

```shell
❯ pwd
/Users/jackzhang/src/tmp/roles/geerlingguy.docker

❯ molecule --debug -v test
DEBUG    Validating schema /Users/jackzhang/src/tmp/roles/geerlingguy.docker/molecule/default/molecule.yml.
INFO     default scenario test matrix: dependency, lint, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO     Performing prerun...
INFO     Set ANSIBLE_LIBRARY=/Users/jackzhang/.cache/ansible-compat/71365d/modules:/Users/jackzhang/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/Users/jackzhang/.cache/ansible-compat/71365d/collections:/Users/jackzhang/.ansible/collections:/usr/share/ansible/collections
INFO     Set ANSIBLE_ROLES_PATH=/Users/jackzhang/.cache/ansible-compat/71365d/roles:/Users/jackzhang/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Using /Users/jackzhang/.ansible/roles/geerlingguy.docker symlink to current repository in order to enable Ansible to find the role using its expected full name.
INFO     Running default > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running default > lint
INFO     Lint is disabled.
INFO     Running default > cleanup
WARNING  Skipping, cleanup playbook not configured.
```

I'm guessing there's some weird issue with molecule resolving system paths that was introduced in 3.6.1 (that was not present in 3.6.0 and below).

Steps to reproduce on my own system:

```
 10:11:26 ~/Downloads $ git clone git@github.com:geerlingguy/ansible-role-containerd    
Cloning into 'ansible-role-containerd'...

 10:11:26 ~/Downloads $ cd ansible-role-containerd 

 10:11:29 ~/Downloads/ansible-role-containerd $ molecule test
INFO     default scenario test matrix: dependency, lint, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO     Performing prerun...
INFO     Set ANSIBLE_LIBRARY=/Users/jgeerling/.cache/ansible-compat/f490ee/modules:/Users/jgeerling/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/Users/jgeerling/.cache/ansible-compat/f490ee/collections:/Users/jgeerling/.ansible/collections:/usr/share/ansible/collections
INFO     Set ANSIBLE_ROLES_PATH=/Users/jgeerling/.cache/ansible-compat/f490ee/roles:/Users/jgeerling/Dropbox/VMs/roles
Traceback (most recent call last):
  File "/opt/homebrew/bin/molecule", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/molecule/command/test.py", line 161, in test
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/opt/homebrew/lib/python3.9/site-packages/molecule/command/base.py", line 111, in execute_cmdline_scenarios
    scenario.config.runtime.prepare_environment(install_local=True)
  File "/opt/homebrew/lib/python3.9/site-packages/ansible_compat/runtime.py", line 334, in prepare_environment
    self._install_galaxy_role(self.project_dir, ignore_errors=True)
  File "/opt/homebrew/lib/python3.9/site-packages/ansible_compat/runtime.py", line 494, in _install_galaxy_role
    if not exists or os.readlink(link_path) != str(target):
OSError: [Errno 22] Invalid argument: '/Users/jgeerling/Dropbox/VMs/roles/geerlingguy.containerd'
```

The folder `/Users/jgeerling/Dropbox/VMs/roles/geerlingguy.containerd` exists, and is readable. My Ansible configuration has the following roles_path:

```
roles_path = ~/Dropbox/VMs/roles
```

And so my guess is something inside `_install_galaxy_role` is checking the path for `geerlingguy.containerd` (in this case—but same if I run the test in another role on my system), and for some reason bailing out (on `if not exists or os.readlink(link_path) != str(target):`).
I wonder if this is related to: https://github.com/ansible/ansible-compat/issues/97
Yes, it is similar to [https://github.com/ansible/ansible-compat/issues/97](https://github.com/ansible/ansible-compat/issues/97).

## root cause

The root cause is there is a conflict between the role installed by `ansible-galaxy` and the role generated by `molecule`, which are in the same role path.

Molecule tried to create a symlink under the role path, but after the `ansible-galaxy` installation, `geerlingguy.containerd` is created as a directory.

No matter customized role path is defined,  this issue can be reproduced by

```shell
$ cd ~/Downloads
$ ansible-galaxy role install geerlingguy.containerd
$ git clone git@github.com:geerlingguy/ansible-role-containerd
$ cd ansible-role-containerd
$ molecule test
```

## workaround

Define a different [role path environment variable0[https://docs.ansible.com/ansible/latest/reference_appendices/config.html#envvar-ANSIBLE_ROLES_PATH] before running `molecule` if both the role installed by `ansible-galaxy` or the role generated by `molecule`.

```shell
$ export ANSIBLE_ROLES_PATH=~/.ansible/molecule/roles
$ cd ~/Downloads/ansible-role-containerd
$ molecule test
```

## follow up

I think this scenario need to be considered. It is useful if role developers want to install a stable role from galaxy and develop/test the role by molecule. I would try to create a PR to fix it.
Thanks—the bigger question I have then is what changed in molecule 3.6.1 that caused this issue—or was it an issue with the compat library version that's included? The breakup of the two projects has made it a lot harder to try tracking down issues in code.
This is related to #3464  and [#3409 Fix prepare_environment call for role installs](https://github.com/ansible-community/molecule/pull/3409/files#r802489585)

In 3.6.1, `install_local` is set to `True`, and there will be a conflict mentioned before.

If the `install_local` is set to `False`, the symlink would not be created.

I'll try to fix it, but it may take some time.

Resolving patch:
diff --git a/src/molecule/provisioner/ansible.py b/src/molecule/provisioner/ansible.py
--- a/src/molecule/provisioner/ansible.py
+++ b/src/molecule/provisioner/ansible.py
@@ -205,7 +205,7 @@ class Ansible(base.Base):
     ::
 
         ANSIBLE_ROLES_PATH:
-          $ephemeral_directory/roles/:$project_directory/../:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
+          $runtime_cache_dir/roles:$ephemeral_directory/roles/:$project_directory/../:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
         ANSIBLE_LIBRARY:
           $ephemeral_directory/modules/:$project_directory/library/:~/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
         ANSIBLE_FILTER_PLUGINS:
@@ -419,9 +419,14 @@ def default_env(self):
         # top of the collection_path_list. This prevents dependency commands
         # from installing dependencies to user list of collections.
         collections_path_list = [
+            util.abs_path(
+                os.path.join(
+                    self._config.scenario.config.runtime.cache_dir, "collections"
+                )
+            ),
             util.abs_path(
                 os.path.join(self._config.scenario.ephemeral_directory, "collections")
-            )
+            ),
         ]
         if collection_indicator in self._config.project_directory:
             collection_path, right = self._config.project_directory.rsplit(
@@ -437,28 +442,30 @@ def default_env(self):
                 "/etc/ansible/collections",
             ]
         )
+
+        roles_path_list = [
+            util.abs_path(
+                os.path.join(self._config.scenario.config.runtime.cache_dir, "roles")
+            ),
+            util.abs_path(
+                os.path.join(self._config.scenario.ephemeral_directory, "roles")
+            ),
+            util.abs_path(os.path.join(self._config.project_directory, os.path.pardir)),
+            util.abs_path(os.path.join(os.path.expanduser("~"), ".ansible", "roles")),
+            "/usr/share/ansible/roles",
+            "/etc/ansible/roles",
+        ]
+
+        if os.environ.get("ANSIBLE_ROLES_PATH", ""):
+            roles_path_list.extend(
+                list(map(util.abs_path, os.environ["ANSIBLE_ROLES_PATH"].split(":")))
+            )
+
         env = util.merge_dicts(
             os.environ,
             {
                 "ANSIBLE_CONFIG": self._config.provisioner.config_file,
-                "ANSIBLE_ROLES_PATH": ":".join(
-                    [
-                        util.abs_path(
-                            os.path.join(
-                                self._config.scenario.ephemeral_directory, "roles"
-                            )
-                        ),
-                        util.abs_path(
-                            os.path.join(self._config.project_directory, os.path.pardir)
-                        ),
-                        util.abs_path(
-                            os.path.join(os.path.expanduser("~"), ".ansible", "roles")
-                        ),
-                        "/usr/share/ansible/roles",
-                        "/etc/ansible/roles",
-                        *os.environ.get("ANSIBLE_ROLES_PATH", "").split(":"),
-                    ]
-                ),
+                "ANSIBLE_ROLES_PATH": ":".join(roles_path_list),
                 self._config.ansible_collections_path: ":".join(collections_path_list),
                 "ANSIBLE_LIBRARY": ":".join(self._get_modules_directories()),
                 "ANSIBLE_FILTER_PLUGINS": ":".join(
diff --git a/src/molecule/provisioner/ansible_playbook.py b/src/molecule/provisioner/ansible_playbook.py
--- a/src/molecule/provisioner/ansible_playbook.py
+++ b/src/molecule/provisioner/ansible_playbook.py
@@ -51,9 +51,7 @@ def __init__(self, playbook, config, verify=False):
                 self._config.verifier.env, self._config.config["verifier"]["env"]
             )
         else:
-            self._env = util.merge_dicts(
-                self._config.provisioner.env, self._config.runtime.environ
-            )
+            self._env = self._config.provisioner.env
 
     def bake(self):
         """


Test patch:
diff --git a/src/molecule/test/unit/provisioner/test_ansible.py b/src/molecule/test/unit/provisioner/test_ansible.py
--- a/src/molecule/test/unit/provisioner/test_ansible.py
+++ b/src/molecule/test/unit/provisioner/test_ansible.py
@@ -204,11 +204,10 @@ def test_env_property(_instance):
     "config_instance", ["_provisioner_section_data"], indirect=True
 )
 def test_env_appends_env_property(_instance):
+    os.environ["ANSIBLE_ROLES_PATH"] = ""
 
-    # molecule could decide to add extra paths, so we only want to check
-    # that those that we need are kept inside the list
-    roles_path_list = _instance.env["ANSIBLE_ROLES_PATH"].split(":")
-    for x in [
+    expected = [
+        util.abs_path(os.path.join(_instance._config.runtime.cache_dir, "roles")),
         util.abs_path(
             os.path.join(_instance._config.scenario.ephemeral_directory, "roles")
         ),
@@ -219,8 +218,14 @@ def test_env_appends_env_property(_instance):
         "/usr/share/ansible/roles",
         "/etc/ansible/roles",
         util.abs_path(os.path.join(_instance._config.scenario.directory, "foo", "bar")),
-    ]:
-        assert x in roles_path_list
+    ]
+
+    # molecule could decide to add extra paths, so we only want to check
+    # that those that we need are kept inside the list with exact order
+
+    roles_path_list = _instance.env["ANSIBLE_ROLES_PATH"].split(":")
+
+    assert roles_path_list == expected
 
     x = _instance._get_modules_directories()
     x.append(
@@ -247,6 +252,36 @@ def test_env_appends_env_property(_instance):
     assert x == _instance.env["ANSIBLE_FILTER_PLUGINS"].split(":")
 
 
+@pytest.mark.parametrize(
+    "config_instance", ["_provisioner_section_data"], indirect=True
+)
+def test_env_appends_env_property_with_os_env(_instance):
+    os.environ["ANSIBLE_ROLES_PATH"] = "/foo/bar:/foo/baz"
+
+    expected = [
+        util.abs_path(os.path.join(_instance._config.runtime.cache_dir, "roles")),
+        util.abs_path(
+            os.path.join(_instance._config.scenario.ephemeral_directory, "roles")
+        ),
+        util.abs_path(
+            os.path.join(_instance._config.project_directory, os.path.pardir)
+        ),
+        util.abs_path(os.path.join(os.path.expanduser("~"), ".ansible", "roles")),
+        "/usr/share/ansible/roles",
+        "/etc/ansible/roles",
+        "/foo/bar",
+        "/foo/baz",
+        util.abs_path(os.path.join(_instance._config.scenario.directory, "foo", "bar")),
+    ]
+
+    # molecule could decide to add extra paths, so we only want to check
+    # that those that we need are kept inside the list
+
+    roles_path_list = _instance.env["ANSIBLE_ROLES_PATH"].split(":")
+
+    assert roles_path_list == expected
+
+
 @pytest.mark.parametrize(
     "config_instance", ["_provisioner_section_data"], indirect=True
 )