Repository: ansible/ansible
Pull request: fix loading vars_plugins in roles
##### SUMMARY

Fixes #82239

Adding a new role dir to the plugin loader clears the plugin loader cache, but the variable `cached_vars_plugin_order` in ansible.vars.plugins is never reset, preventing new plugins from being found.

Remove cache in the middle and use the vars plugin loader cache as the load-order. To make this possible, the cache keys now include non-stateless vars plugins (the value is a None, unresolved PluginLoadContext() tuple to be easily distinguished from stateless plugins).

##### ISSUE TYPE

<!--- Pick one below and delete the rest -->

- Bugfix Pull Request

##### ADDITIONAL INFORMATION

<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->

<!--- Paste verbatim command output below, e.g. before and after your change -->

```paste below

```


Problem statement:
Ansible 2.16 cannot load a vars plugin from role anymore
### Summary

More specifically, I am using gekmihesg.openwrt to manage a number of OpenWRT routers (which do not have python support) using distro version(s) of ansible.

With upgrade to Fedora 39, I got ansible-core 2.16.0~rc1, release 1.fc39 which I suppose is something very close to the current development version. This version fails to pick up the openwrt vars_plugin at all, and I had limited luck in debugging what happens. Moving on to poetry install ansible instead, I notice I get 'ansible [core 2.15.6]' and everything works just fine (after I remove the exception I had introduced for debugging purposes..)

I experimented with various settings of this in ansible.cfg, but it did not change behaviour - the vars_plugins code under the gekmihesg.openwrt directory appears to never get called:

vars_plugins_enabled = ansible.builtin.host_group_vars,gekmihesg.openwrt

Anyway, whether I am configuring something wrong or need something extra w/version 2.16 was kind of frustrating to figure out so I thought you might want to know about it so that at least the documentation can be improved.

### Issue Type

Bug Report

### Component Name

vars_plugins

### Ansible Version

```console
$ ansible --version
ansible [core 2.16.0rc1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/knuto/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.12/site-packages/ansible
  ansible collection location = /home/knuto/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.0 (main, Oct  2 2023, 00:00:00) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
```


### Configuration

```console
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CONFIG_FILE() = /etc/ansible/ansible.cfg
EDITOR(env: EDITOR) = emacs
PAGER(env: PAGER) = less
```


### OS / Environment

cat /etc/fedora-release
Fedora release 39 (Thirty Nine)

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)

```


### Expected Results

I expect 'uci' rules from the OpenWRT package to execute as normal (eg. not invoking python on the remote side but run via the shell script wrappers in gekmihesg.openwrt)


### Actual Results

```console
The first rule that should have been patched by the plugin to run via the wrapper scripts instead of via python fails to pick up any interpreter (because no one exists..):

[WARNING]: No python interpreters found for host r120 (tried ['python3.12', 'python3.11', 'python3.10', 'python3.9', 'python3.8', 'python3.7', 'python3.6', '/usr/bin/python3', '/usr/libexec/platform-
python', 'python2.7', '/usr/bin/python', 'python'])
fatal: [r120]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "module_stderr": "OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023\r\ndebug1: Reading configuration data /home/knuto/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for 'final all' host 192.168.2.1 originally 192.168.2.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 192.168.2.1 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /home/knuto/.ssh/config\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug3: kex names ok: [diffie-hellman-group1-sha1]\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for 'final all' host 192.168.2.1 originally 192.168.2.1\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 146502\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\n/bin/sh: /usr/bin/python: not found\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 127\r\n",
    "module_stdout": "",
    "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
    "rc": 127
}
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct

Hints from issue discussion:
Files identified in the description:

None

If these files are incorrect, please update the `component name` section of the description or use the [component](https://github.com/ansible/ansibotmini#ansibotmini) bot command.

<!--- boilerplate: components_banner --->

> Moving on to poetry install ansible instead, I notice I get 'ansible [core 2.15.6]' and everything works just fine (after I remove the exception I had introduced for debugging purposes..)

Note that ansible-core 2.15 does not support Fedora 39 and Python 3.12 which is why we have ansible-core-2.16.0~rc1 packaged (the Fedora 2.16.0 release build is also [coming down the pipe](https://bodhi.fedoraproject.org/updates/FEDORA-2023-0fec669bd8)).

> the vars_plugins code under the gekmihesg.openwrt directory appears to never get called

Are you sure the plugin is actually enabled and properly configured? Your `ansible-config dump --only-changed -t all` output does not show that it's enabled. If the vars plugin is indeed enabled, it's possible that whatever [brittle monkeypatching](https://github.com/gekmihesg/ansible-openwrt/blob/master/vars_plugins/monkeypatch.py) it's doing is not compatible with ansible-core 2.16. In that case, this bug should be reported to that project and not to ansible-core.
> Are you sure the plugin is actually enabled and properly configured? Your `ansible-config dump --only-changed -t all` 
output does not show that it's enabled. If the vars plugin is indeed enabled, it's possible that whatever [brittle monkeypatching](https://github.com/gekmihesg/ansible-openwrt/blob/master/vars_plugins/monkeypatch.py) it's doing is not compatible with ansible-core 2.16. In that case, this bug should be reported to that project and not to ansible-core.

Thanks for the quick followup.
I am sorry, but it seems I must have made some mistake when running the configuration dump.
With what I suppose is the right configuration file line for the vars plugins:

`[defaults]
vars_plugins_enabled = ansible.builtin.host_group_vars,gekmihesg.openwrt`

I get this from `ansible-config dump --only-changed -t all`:

```
CONFIG_FILE() = /home/knuto/src/energima/infrastructure/ansible.cfg
DEFAULT_HOST_LIST(/home/knuto/src/energima/infrastructure/ansible.cfg) = ['/home/knuto/src/energima/infrastructure/hosts']
DISPLAY_SKIPPED_HOSTS(/home/knuto/src/energima/infrastructure/ansible.cfg) = False
EDITOR(env: EDITOR) = emacs
INVENTORY_ENABLED(/home/knuto/src/energima/infrastructure/ansible.cfg) = ['ini']
PAGER(env: PAGER) = less
USE_PERSISTENT_CONNECTIONS(/home/knuto/src/energima/infrastructure/ansible.cfg) = True
VARIABLE_PLUGINS_ENABLED(/home/knuto/src/energima/infrastructure/ansible.cfg) = ['ansible.builtin.host_group_vars', 'gekmihesg.openwrt']

CALLBACK:
========

default:
_______
display_skipped_hosts(/home/knuto/src/energima/infrastructure/ansible.cfg) = False

CONNECTION:
==========

paramiko_ssh:
____________
ssh_args(/home/knuto/src/energima/infrastructure/ansible.cfg) = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s
use_persistent_connections(/home/knuto/src/energima/infrastructure/ansible.cfg) = True

ssh:
CONFIG_FILE() = /home/knuto/src/energima/infrastructure/ansible.cfg
DEFAULT_HOST_LIST(/home/knuto/src/energima/infrastructure/ansible.cfg) = ['/home/knuto/src/energima/infrastructure/hosts']
DISPLAY_SKIPPED_HOSTS(/home/knuto/src/energima/infrastructure/ansible.cfg) = False
EDITOR(env: EDITOR) = emacs
INVENTORY_ENABLED(/home/knuto/src/energima/infrastructure/ansible.cfg) = ['ini']
PAGER(env: PAGER) = less
USE_PERSISTENT_CONNECTIONS(/home/knuto/src/energima/infrastructure/ansible.cfg) = True
VARIABLE_PLUGINS_ENABLED(/home/knuto/src/energima/infrastructure/ansible.cfg) = ['ansible.builtin.host_group_vars', 'gekmihesg.openwrt']

CALLBACK:
========

default:
_______
display_skipped_hosts(/home/knuto/src/energima/infrastructure/ansible.cfg) = False

CONNECTION:
==========

paramiko_ssh:
____________
ssh_args(/home/knuto/src/energima/infrastructure/ansible.cfg) = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s
use_persistent_connections(/home/knuto/src/energima/infrastructure/ansible.cfg) = True

ssh:
___
ssh_args(/home/knuto/src/energima/infrastructure/ansible.cfg) = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s

```
What would be good would be if the VARIABLE_PLUGINS_ENABLED variable was somehow checked. Maybe I am just using the wrong naming or otherwise miss something in the setup? It took me a while just to realize that I needed to also provide `ansible.builtin.host_group_vars` in the config setting, otherwise I would get a different, much earlier error.
And it is still not clear to me whether just `gekmihesg.openwrt` is the right way to reference the vars plugin since it really references a role and maybe not the plugin itself?

> If the vars plugin is indeed enabled, it's possible that whatever [brittle monkeypatching](https://github.com/gekmihesg/ansible-openwrt/blob/master/vars_plugins/monkeypatch.py) it's doing is not compatible with ansible-core 2.16. In that case, this bug should be reported to that project and not to ansible-core.

If the plugin gets enabled, I would expect the python code in the link would be evaluated, but no tricks I could think of (such as adding a breakpoint() or a syntax error in the plugin would yield any effect?

Besides that, I agree with the brittleness, and I believe so does the author, but apparently no good alternatives arised from the initial discussion around this. Is there a better approach to handle targets which cannot run python? The plugin has been really useful and blends in well with other (python supported) targets. I maintain another set of routers with custom built Linux images, there I have so far had to rely entirely on raw rules, a much less satisfactory solution. Anyway, this is maybe a separate topic.



First `gekmihesg.openwrt`  is just a collection name, not a valid reference to the plugin itself, much less to a plugin inside a role. 

The issue might stem from this bugfix #79945.  Which is now probably happening too soon for a vars plugin in a role to be loaded.

As for the monkey patch ... I would really just use the `openwrt_module` actions instead of tryinig to hijack and rename 'normal modules' ..specially since not all have a openwrt.sh version. Also, instead of a vars plugin, I would use an overriden `normal.py` action plugin to do so, would not even need a monkey patch, just rename the module before calling `_configure`.

Ansible already supports using 'non python modules' and even dealing with multple interpreters ( `ansible_perl_interpreter`, `ansible_bash_interpreter`, etc) , you can even 'hijack' the module names by just adding them to `library/` or `action_plugins/` (for things like copy and template), it does not support simple renaming in that way, you can kind of do it with collections, but that is not something i recommend.
A quick workaround would be to create an actual collection and move the vars plugin there, then you could reference it correctly in config (`myns.myopenwrt.monkeypatch`).

Commit message: remove extra exception handling

Patch to review:
diff --git a/changelogs/fragments/fix-vars-plugins-in-roles.yml b/changelogs/fragments/fix-vars-plugins-in-roles.yml
new file mode 100644
index 00000000000000..b64d586b9ef5ff
--- /dev/null
+++ b/changelogs/fragments/fix-vars-plugins-in-roles.yml
@@ -0,0 +1,2 @@
+bugfixes:
+  - Fix loading vars_plugins in roles (https://github.com/ansible/ansible/issues/82239).
diff --git a/lib/ansible/plugins/loader.py b/lib/ansible/plugins/loader.py
index 90ff17b198a512..1a5043aa1a02fc 100644
--- a/lib/ansible/plugins/loader.py
+++ b/lib/ansible/plugins/loader.py
@@ -13,7 +13,7 @@
 import sys
 import warnings
 
-from collections import defaultdict, namedtuple
+from collections import defaultdict, namedtuple, OrderedDict
 from importlib import import_module
 from traceback import format_exc
 
@@ -237,10 +237,7 @@ def __init__(self, class_name, package, config, subdir, aliases=None, required_b
         self._module_cache = MODULE_CACHE[class_name]
         self._paths = PATH_CACHE[class_name]
         self._plugin_path_cache = PLUGIN_PATH_CACHE[class_name]
-        try:
-            self._plugin_instance_cache = {} if self.type == 'vars' else None
-        except ValueError:
-            self._plugin_instance_cache = None
+        self._plugin_instance_cache = OrderedDict() if self.subdir == 'vars_plugins' else None
 
         self._searched_paths = set()
 
@@ -265,7 +262,7 @@ def _clear_caches(self):
             self._module_cache = MODULE_CACHE[self.class_name]
             self._paths = PATH_CACHE[self.class_name]
             self._plugin_path_cache = PLUGIN_PATH_CACHE[self.class_name]
-            self._plugin_instance_cache = {} if self.type == 'vars' else None
+            self._plugin_instance_cache = OrderedDict() if self.subdir == 'vars_plugins' else None
             self._searched_paths = set()
 
     def __setstate__(self, data):
@@ -871,7 +868,7 @@ def get_with_context(self, name, *args, **kwargs):
         if name in self.aliases:
             name = self.aliases[name]
 
-        if self._plugin_instance_cache and (cached_load_result := self._plugin_instance_cache.get(name)):
+        if self._plugin_instance_cache and (cached_load_result := self._plugin_instance_cache.get(name)) and cached_load_result[1].resolved:
             # Resolving the FQCN is slow, even if we've passed in the resolved FQCN.
             # Short-circuit here if we've previously resolved this name.
             # This will need to be restricted if non-vars plugins start using the cache, since
@@ -888,7 +885,7 @@ def get_with_context(self, name, *args, **kwargs):
             fq_name = '.'.join((plugin_load_context.plugin_resolved_collection, fq_name))
         resolved_type_name = plugin_load_context.plugin_resolved_name
         path = plugin_load_context.plugin_resolved_path
-        if self._plugin_instance_cache and (cached_load_result := self._plugin_instance_cache.get(fq_name)):
+        if self._plugin_instance_cache and (cached_load_result := self._plugin_instance_cache.get(fq_name)) and cached_load_result[1].resolved:
             # This is unused by vars plugins, but it's here in case the instance cache expands to other plugin types.
             # We get here if we've seen this plugin before, but it wasn't called with the resolved FQCN.
             return get_with_context_result(*cached_load_result)
@@ -934,8 +931,9 @@ def get_with_context(self, name, *args, **kwargs):
 
         self._update_object(obj, resolved_type_name, path, redirected_names, fq_name)
         if self._plugin_instance_cache is not None and getattr(obj, 'is_stateless', False):
-            # store under both the originally requested name and the resolved FQ name
-            self._plugin_instance_cache[name] = self._plugin_instance_cache[fq_name] = (obj, plugin_load_context)
+            self._plugin_instance_cache[fq_name] = (obj, plugin_load_context)
+        elif self._plugin_instance_cache is not None:
+            self._plugin_instance_cache[fq_name] = (None, PluginLoadContext())
         return get_with_context_result(obj, plugin_load_context)
 
     def _display_plugin_load(self, class_name, name, searched_paths, path, found_in_cache=None, class_only=None):
@@ -1041,9 +1039,9 @@ def all(self, *args, **kwargs):
             else:
                 fqcn = f"ansible.builtin.{basename}"
 
-            if self._plugin_instance_cache is not None and fqcn in self._plugin_instance_cache:
+            if fqcn in (self._plugin_instance_cache or {}) and self._plugin_instance_cache[fqcn][1].resolved:
                 # Here just in case, but we don't call all() multiple times for vars plugins, so this should not be used.
-                yield self._plugin_instance_cache[basename][0]
+                yield self._plugin_instance_cache[fqcn][0]
                 continue
 
             if path not in self._module_cache:
@@ -1095,9 +1093,17 @@ def all(self, *args, **kwargs):
 
             self._update_object(obj, basename, path, resolved=fqcn)
 
-            if self._plugin_instance_cache is not None and fqcn not in self._plugin_instance_cache:
-                # Use get_with_context to cache the plugin the first time we see it.
-                self.get_with_context(fqcn)[0]
+            if self._plugin_instance_cache is not None:
+                needs_enabled = False
+                if hasattr(obj, 'REQUIRES_ENABLED'):
+                    needs_enabled = obj.REQUIRES_ENABLED
+                elif hasattr(obj, 'REQUIRES_WHITELIST'):
+                    needs_enabled = obj.REQUIRES_WHITELIST
+                    display.deprecated("The VarsModule class variable 'REQUIRES_WHITELIST' is deprecated. "
+                                       "Use 'REQUIRES_ENABLED' instead.", version=2.18)
+                if not needs_enabled:
+                    # Use get_with_context to cache the plugin the first time we see it.
+                    self.get_with_context(fqcn)[0]
 
             yield obj
 
diff --git a/lib/ansible/vars/plugins.py b/lib/ansible/vars/plugins.py
index 37624d9de78809..23baa5a46775d4 100644
--- a/lib/ansible/vars/plugins.py
+++ b/lib/ansible/vars/plugins.py
@@ -16,41 +16,14 @@
 
 display = Display()
 
-cached_vars_plugin_order = None
 
-
-def _load_vars_plugins_order():
+def _prime_vars_loader():
     # find 3rd party legacy vars plugins once, and look them up by name subsequently
-    auto = []
-    for auto_run_plugin in vars_loader.all(class_only=True):
-        needs_enabled = False
-        if hasattr(auto_run_plugin, 'REQUIRES_ENABLED'):
-            needs_enabled = auto_run_plugin.REQUIRES_ENABLED
-        elif hasattr(auto_run_plugin, 'REQUIRES_WHITELIST'):
-            needs_enabled = auto_run_plugin.REQUIRES_WHITELIST
-            display.deprecated("The VarsModule class variable 'REQUIRES_WHITELIST' is deprecated. "
-                               "Use 'REQUIRES_ENABLED' instead.", version=2.18)
-        if needs_enabled:
-            continue
-        auto.append(auto_run_plugin._load_name)
-
-    # find enabled plugins once so we can look them up by resolved fqcn subsequently
-    enabled = []
+    list(vars_loader.all(class_only=True))
     for plugin_name in C.VARIABLE_PLUGINS_ENABLED:
-        if (plugin := vars_loader.get(plugin_name)) is None:
-            enabled.append(plugin_name)
-        else:
-            collection = '.' in plugin.ansible_name and not plugin.ansible_name.startswith('ansible.builtin.')
-            # Warn if a collection plugin has REQUIRES_ENABLED because it has no effect.
-            if collection and (hasattr(plugin, 'REQUIRES_ENABLED') or hasattr(plugin, 'REQUIRES_WHITELIST')):
-                display.warning(
-                    "Vars plugins in collections must be enabled to be loaded, REQUIRES_ENABLED is not supported. "
-                    "This should be removed from the plugin %s." % plugin.ansible_name
-                )
-            enabled.append(plugin.ansible_name)
-
-    global cached_vars_plugin_order
-    cached_vars_plugin_order = auto + enabled
+        if not plugin_name:
+            continue
+        vars_loader.get(plugin_name)
 
 
 def get_plugin_vars(loader, plugin, path, entities):
@@ -106,16 +79,23 @@ def _plugin_should_run(plugin, stage):
 
 
 def get_vars_from_path(loader, path, entities, stage):
-
     data = {}
+    # FIXME: _prime_vars_loader shouldn't run repeatedly if there are no plugins
+    if not vars_loader._plugin_instance_cache:
+        _prime_vars_loader()
 
-    if cached_vars_plugin_order is None:
-        _load_vars_plugins_order()
-
-    for plugin_name in cached_vars_plugin_order:
+    for plugin_name in vars_loader._plugin_instance_cache:
         if (plugin := vars_loader.get(plugin_name)) is None:
             continue
 
+        collection = '.' in plugin.ansible_name and not plugin.ansible_name.startswith('ansible.builtin.')
+        # Warn if a collection plugin has REQUIRES_ENABLED because it has no effect.
+        if collection and (hasattr(plugin, 'REQUIRES_ENABLED') or hasattr(plugin, 'REQUIRES_WHITELIST')):
+            display.warning(
+                "Vars plugins in collections must be enabled to be loaded, REQUIRES_ENABLED is not supported. "
+                "This should be removed from the plugin %s." % plugin.ansible_name
+            )
+
         if not _plugin_should_run(plugin, stage):
             continue
 
diff --git a/test/integration/targets/old_style_vars_plugins/roles/a/tasks/main.yml b/test/integration/targets/old_style_vars_plugins/roles/a/tasks/main.yml
new file mode 100644
index 00000000000000..8e0742a5a5df0d
--- /dev/null
+++ b/test/integration/targets/old_style_vars_plugins/roles/a/tasks/main.yml
@@ -0,0 +1,3 @@
+- assert:
+    that:
+      - auto_role_var is defined
diff --git a/test/integration/targets/old_style_vars_plugins/roles/a/vars_plugins/auto_role_vars.py b/test/integration/targets/old_style_vars_plugins/roles/a/vars_plugins/auto_role_vars.py
new file mode 100644
index 00000000000000..a1cd30d3b04e03
--- /dev/null
+++ b/test/integration/targets/old_style_vars_plugins/roles/a/vars_plugins/auto_role_vars.py
@@ -0,0 +1,11 @@
+from __future__ import annotations
+
+from ansible.plugins.vars import BaseVarsPlugin
+
+
+class VarsModule(BaseVarsPlugin):
+    # Implicitly
+    # REQUIRES_ENABLED = False
+
+    def get_vars(self, loader, path, entities):
+        return {'auto_role_var': True}
diff --git a/test/integration/targets/old_style_vars_plugins/runme.sh b/test/integration/targets/old_style_vars_plugins/runme.sh
index 71275b8ac47b95..9f416235bb0895 100755
--- a/test/integration/targets/old_style_vars_plugins/runme.sh
+++ b/test/integration/targets/old_style_vars_plugins/runme.sh
@@ -46,3 +46,5 @@ ANSIBLE_DEBUG=True ansible-playbook test_task_vars.yml > out.txt
 [ "$(grep -c "Loading VarsModule 'host_group_vars'" out.txt)" -eq 1 ]
 [ "$(grep -c "Loading VarsModule 'require_enabled'" out.txt)" -lt 3 ]
 [ "$(grep -c "Loading VarsModule 'auto_enabled'" out.txt)" -gt 50 ]
+
+ansible localhost -m include_role -a 'name=a' "$@"


Code review comments:
diff_hunk: @@ -13,7 +13,7 @@
 import sys
 import warnings
 
-from collections import defaultdict, namedtuple
+from collections import defaultdict, namedtuple, OrderedDict
path: lib/ansible/plugins/loader.py
text: @user1:
Is there a specific reason `OrderedDict` is needed?  Insertion order of dicts has been official since py3.7.