Repository: ansible/ansible
Pull request: Add some documentation about plugins in collections
##### SUMMARY
Fixes #62231

Add some documentation about things that require FQCN, limitations on inventory caching, and code sharing.

Vars plugin documentation is in #61078. Vars plugins in collections are ignored prior to #61078.

##### ISSUE TYPE
- Docs Pull Request

Problem statement:
Unclear how to use inventory plugins in collections
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
I've been unable to get inventory plugins working inside of collections. The inventory plugin in ansible/ansible always takes precedence.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
collections

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
devel
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below

```

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->


##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->
I've got a inventory.gcp.yml file that I run with `ansible-inventory --list -i inventory.gcp.yml`. I've got a collection called google.cloud installed into my collections path. Here are the various things I've tried.

```yaml
plugin: gcp_compute
...
```
Result: Runs the ansible/ansible inventory plugin.

```yaml
collections:
- google.cloud
plugin: gcp_compute
...
```
Result: Runs the ansible/ansible inventory plugin

```yaml
plugin: google.cloud.gcp_compute
...
```

Result: Plugin not found, errors out.

(I've also tried with `google.cloud.plugins.inventory.gcp_compute` and `google.cloud.inventory.gcp_compute` to no avail)
<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->


##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes -->
```paste below

```

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/module_utils/common/collections.py`](https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/common/collections.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
did you whitelist it?
I wasn’t aware of any whitelisting. How do you whitelist plugins?

On Thu, Sep 12, 2019 at 4:33 PM Brian Coca <notifications@github.com> wrote:

> did you whitelist it?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/62231?email_source=notifications&email_token=AAKDVZSFS5MUHZDKDH7CTZ3QJLGUBA5CNFSM4IWKLHS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6TRTZY#issuecomment-531044839>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKDVZTTU3NHC5IV2KSIXW3QJLGUBANCNFSM4IWKLHSQ>
> .
>

@rambleraptor Make sure the NAME in your inventory plugin in the collection uses the FQCN - `NAME = 'google.cloud.gcp_compute'` and that your config file uses it too `plugin: google.cloud.gcp_compute`.

If you're using a collection adjacent to a playbook, use the `--playbook-dir` flag with `ansible-inventory`. If you're using a general collection make sure the collection is in an expected path https://github.com/ansible/ansible/blob/devel/lib/ansible/config/base.yml#L218-L226.

Either `auto` (which is whitelisted by default) or the specific plugin should be whitelisted, e.g.:
```
# ansible.cfg
[inventory]
enable_plugins = google.cloud.gcp_compute
```

I added a bit of documentation in #62465. If you have any feedback it would be appreciated.
The documentation is fantastic. Exactly what I was looking for. Thanks so much!

Commit message: Clarify plugins that don't work in collections and make sure to use FQCN in examples where necessary

Patch to review:
diff --git a/docs/docsite/rst/dev_guide/developing_collections.rst b/docs/docsite/rst/dev_guide/developing_collections.rst
index 58999ede94d6c6..f7db43de1a8f1c 100644
--- a/docs/docsite/rst/dev_guide/developing_collections.rst
+++ b/docs/docsite/rst/dev_guide/developing_collections.rst
@@ -72,7 +72,9 @@ The ``ansible-doc`` command requires the fully qualified collection name (FQCN)
 plugins directory
 ------------------
 
-Add a 'per plugin type' specific subdirectory here, including ``module_utils`` which is usable not only by modules, but by any other plugin by using their FQCN. This is a way to distribute modules, lookups, filters, and so on, without having to import a role in every play.
+Add a 'per plugin type' specific subdirectory here, including ``module_utils`` which is usable not only by modules, but by most plugins by using their FQCN. This is a way to distribute modules, lookups, filters, and so on, without having to import a role in every play.
+
+Vars plugins are unsupported in collections. Cache plugins may be used in collections for fact caching, but are not supported for inventory plugins.
 
 module_utils
 ^^^^^^^^^^^^
@@ -109,6 +111,11 @@ In the Python example the ``module_util`` in question is called ``qradar`` such
         not_rest_data_keys=['state']
     )
 
+Note that importing something from an ``__init__.py`` file requires using the file name:
+
+.. code-block:: python
+
+    from ansible_collections.namespace.collection_name.plugins.callback.__init__ import CustomBaseClass
 
 In the PowerShell example the ``module_util`` in question is called ``hyperv`` such that the FCQN is
 ``ansible_example.community.plugins.module_utils.hyperv``:
diff --git a/docs/docsite/rst/dev_guide/developing_inventory.rst b/docs/docsite/rst/dev_guide/developing_inventory.rst
index 13c880f1847587..e93787014f82e9 100644
--- a/docs/docsite/rst/dev_guide/developing_inventory.rst
+++ b/docs/docsite/rst/dev_guide/developing_inventory.rst
@@ -74,6 +74,8 @@ The first thing you want to do is use the base class:
 
         NAME = 'myplugin'  # used internally by Ansible, it should match the file name but not required
 
+If the inventory plugin is in a collection the NAME should be in the format of 'namespace.collection_name.myplugin'.
+
 This class has a couple of methods each plugin should implement and a few helpers for parsing the inventory source and updating the inventory.
 
 After you have the basic plugin working you might want to to incorporate other features by adding more base classes:
diff --git a/docs/docsite/rst/dev_guide/developing_plugins.rst b/docs/docsite/rst/dev_guide/developing_plugins.rst
index 2bbfe0d2a3f356..b905b7ee154e19 100644
--- a/docs/docsite/rst/dev_guide/developing_plugins.rst
+++ b/docs/docsite/rst/dev_guide/developing_plugins.rst
@@ -150,7 +150,7 @@ see the source code for the `action plugins included with Ansible Core <https://
 Cache plugins
 -------------
 
-Cache plugins store gathered facts and data retrieved by inventory plugins.
+Cache plugins store gathered facts and data retrieved by inventory plugins. Only fact caching is currently supported by cache plugins in collections.
 
 Import cache plugins using the cache_loader so you can use ``self.set_options()`` and ``self.get_option(<option_name>)``. If you import a cache plugin directly in the code base, you can only access options via ``ansible.constants``, and you break the cache plugin's ability to be used by an inventory plugin.
 
@@ -246,7 +246,7 @@ but with an extra option so you can see how configuration works in Ansible versi
       """
       CALLBACK_VERSION = 2.0
       CALLBACK_TYPE = 'aggregate'
-      CALLBACK_NAME = 'timer'
+      CALLBACK_NAME = 'namespace.collection_name.timer'
 
       # only needed if you ship it and don't want to enable by default
       CALLBACK_NEEDS_WHITELIST = True
@@ -389,7 +389,7 @@ The following is an example of how this lookup is called::
   ---
   - hosts: all
     vars:
-       contents: "{{ lookup('file', '/etc/foo.txt') }}"
+       contents: "{{ lookup('namespace.collection_name.file', '/etc/foo.txt') }}"
 
     tasks:
 
@@ -416,35 +416,7 @@ For example test plugins, see the source code for the `test plugins included wit
 Vars plugins
 ------------
 
-Vars plugins inject additional variable data into Ansible runs that did not come from an inventory source, playbook, or command line. Playbook constructs like 'host_vars' and 'group_vars' work using vars plugins.
-
-Vars plugins were partially implemented in Ansible 2.0 and rewritten to be fully implemented starting with Ansible 2.4.
-
-Older plugins used a ``run`` method as their main body/work:
-
-.. code-block:: python
-
-    def run(self, name, vault_password=None):
-        pass # your code goes here
-
-
-Ansible 2.0 did not pass passwords to older plugins, so vaults were unavailable.
-Most of the work now  happens in the ``get_vars`` method which is called from the VariableManager when needed.
-
-.. code-block:: python
-
-    def get_vars(self, loader, path, entities):
-        pass # your code goes here
-
-The parameters are:
-
- * loader: Ansible's DataLoader. The DataLoader can read files, auto-load JSON/YAML and decrypt vaulted data, and cache read files.
- * path: this is 'directory data' for every inventory source and the current play's playbook directory, so they can search for data in reference to them. ``get_vars`` will be called at least once per available path.
- * entities: these are host or group names that are pertinent to the variables needed. The plugin will get called once for hosts and again for groups.
-
-This ``get vars`` method just needs to return a dictionary structure with the variables.
-
-Since Ansible version 2.4, vars plugins only execute as needed when preparing to execute a task. This avoids the costly 'always execute' behavior that occurred during inventory construction in older versions of Ansible.
+Vars plugins are unsupported by collections in Ansible version 2.9.
 
 For example vars plugins, see the source code for the `vars plugins included with Ansible Core
 <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/vars>`_.
diff --git a/docs/docsite/rst/plugins/cache.rst b/docs/docsite/rst/plugins/cache.rst
index 66ffbc30e59a55..e6cad2b895d340 100644
--- a/docs/docsite/rst/plugins/cache.rst
+++ b/docs/docsite/rst/plugins/cache.rst
@@ -34,6 +34,13 @@ or in the ``ansible.cfg`` file:
     [defaults]
     fact_caching=redis
 
+If the cache plugin is in a collection use the fully qualified name:
+
+.. code-block:: ini
+
+    [defaults]
+    fact_caching = namespace.collection_name.cache_plugin_name
+
 You will also need to configure other settings specific to each plugin. Consult the individual plugin documentation
 or the Ansible :ref:`configuration <ansible_configuration_settings>` for more details.
 
@@ -43,7 +50,7 @@ A custom cache plugin is enabled by dropping it into a ``cache_plugins`` directo
 Enabling Inventory Cache Plugins
 --------------------------------
 
-Inventory may be cached using a file-based cache plugin (like jsonfile). Check the specific inventory plugin to see if it supports caching.
+Inventory may be cached using a file-based cache plugin (like jsonfile). Check the specific inventory plugin to see if it supports caching. Cache plugins inside a collection are not supported for caching inventory.
 If an inventory-specific cache plugin is not specified Ansible will fall back to caching inventory with the fact cache plugin options.
 
 The inventory cache is disabled by default. You may enable it via environment variable:
diff --git a/docs/docsite/rst/plugins/callback.rst b/docs/docsite/rst/plugins/callback.rst
index 0614a0c9b66f6b..3160eb120fc5af 100644
--- a/docs/docsite/rst/plugins/callback.rst
+++ b/docs/docsite/rst/plugins/callback.rst
@@ -34,7 +34,7 @@ Most callbacks shipped with Ansible are disabled by default and need to be white
 
 .. code-block:: ini
 
-  #callback_whitelist = timer, mail, profile_roles
+  #callback_whitelist = timer, mail, profile_roles, collection_namespace.collection_name.custom_callback
 
 Setting a callback plugin for ``ansible-playbook``
 --------------------------------------------------
diff --git a/docs/docsite/rst/plugins/inventory.rst b/docs/docsite/rst/plugins/inventory.rst
index 0576b35c5413b8..e95a486fe94fca 100644
--- a/docs/docsite/rst/plugins/inventory.rst
+++ b/docs/docsite/rst/plugins/inventory.rst
@@ -31,6 +31,15 @@ This list also establishes the order in which each plugin tries to parse an inve
    [inventory]
    enable_plugins = advanced_host_list, constructed, yaml
 
+The ``auto`` inventory plugin can be used to automatically determines which inventory plugin to use for a YAML configuration file. It can also be used for inventory plugins in a collection.
+
+To whitelist specific inventory plugins in a collection you need to use the fully qualified name:
+
+.. code-block:: ini
+
+   [inventory]
+   enable_plugins = namespace.collection_name.inventory_plugin_name
+
 
 .. _using_inventory:
 
@@ -55,6 +64,12 @@ Or for the openstack plugin the file has to be called ``clouds.yml`` or ``openst
     # clouds.yml or openstack.(yml|yaml)
     plugin: openstack
 
+To use a plugin in a collection provide the fully qualified name:
+
+.. code-block:: yaml
+
+    plugin: namespace.collection_name.inventory_plugin_name
+
 The ``auto`` inventory plugin is enabled by default and works by using the ``plugin`` field to indicate the plugin that should attempt to parse it. You can configure the whitelist/precedence of inventory plugins used to parse source using the `ansible.cfg` ['inventory'] ``enable_plugins`` list. After enabling the plugin and providing any required options, you can view the populated inventory with ``ansible-inventory -i demo.aws_ec2.yml --graph``:
 
 .. code-block:: text
@@ -65,6 +80,8 @@ The ``auto`` inventory plugin is enabled by default and works by using the ``plu
       |  |--ec2-98-765-432-10.compute-1.amazonaws.com
       |--@ungrouped:
 
+If you are using an inventory plugin in a playbook-adjacent collection and want to test your setup with ``ansible-inventory``, you will need to use the --playbook-dir flag.
+
 You can set the default inventory path (via ``inventory`` in the `ansible.cfg` [defaults] section or the :envvar:`ANSIBLE_INVENTORY` environment variable) to your inventory source(s). Now running ``ansible-inventory --graph`` should yield the same output as when you passed your YAML configuration source(s) directly. You can add custom inventory plugins to your plugin path to use in the same way.
 
 Your inventory source might be a directory of inventory configuration files. The constructed inventory plugin only operates on those hosts already in inventory, so you may want the constructed inventory configuration parsed at a particular point (such as last). Ansible parses the directory recursively, alphabetically. You cannot configure the parsing approach, so name your files to make it work predictably. Inventory plugins that extend constructed features directly can work around that restriction by adding constructed options in addition to the inventory plugin options. Otherwise, you can use ``-i`` with multiple sources to impose a specific order, e.g. ``-i demo.aws_ec2.yml -i clouds.yml -i constructed.yml``.
@@ -135,6 +152,8 @@ Here is an example of setting inventory caching with some fact caching defaults
    cache = yes
    cache_connection = /tmp/ansible_inventory
 
+Besides cache plugins shipped with Ansible, cache plugins eligible for caching inventory can also reside in a custom cache plugin path. Cache plugins in collections are not supported yet for inventory.
+
 .. _inventory_plugin_list:
 
 Plugin List


Code review comments:
diff_hunk: @@ -416,35 +416,7 @@ For example test plugins, see the source code for the `test plugins included wit
 Vars plugins
 ------------
 
-Vars plugins inject additional variable data into Ansible runs that did not come from an inventory source, playbook, or command line. Playbook constructs like 'host_vars' and 'group_vars' work using vars plugins.
-
-Vars plugins were partially implemented in Ansible 2.0 and rewritten to be fully implemented starting with Ansible 2.4.
-
-Older plugins used a ``run`` method as their main body/work:
-
-.. code-block:: python
-
-    def run(self, name, vault_password=None):
-        pass # your code goes here
-
-
-Ansible 2.0 did not pass passwords to older plugins, so vaults were unavailable.
-Most of the work now  happens in the ``get_vars`` method which is called from the VariableManager when needed.
-
-.. code-block:: python
-
-    def get_vars(self, loader, path, entities):
-        pass # your code goes here
-
-The parameters are:
-
- * loader: Ansible's DataLoader. The DataLoader can read files, auto-load JSON/YAML and decrypt vaulted data, and cache read files.
- * path: this is 'directory data' for every inventory source and the current play's playbook directory, so they can search for data in reference to them. ``get_vars`` will be called at least once per available path.
- * entities: these are host or group names that are pertinent to the variables needed. The plugin will get called once for hosts and again for groups.
-
-This ``get vars`` method just needs to return a dictionary structure with the variables.
-
-Since Ansible version 2.4, vars plugins only execute as needed when preparing to execute a task. This avoids the costly 'always execute' behavior that occurred during inventory construction in older versions of Ansible.
+Vars plugins are unsupported by collections in Ansible version 2.9.
path: docs/docsite/rst/dev_guide/developing_plugins.rst
text: @user1:
Can you clarify why vars is removed entirely? I see the other PR (#61078) adds this documentation back. The problem I see - this PR can be backported to 2.9, but the other PR cannot. That would leave us with vars docs in 2.9 not having anything but 'not supported in collections'. 

@author:
Oh, oops! I thought this was in the docs/docsite/rst/dev_guide/developing_collections.rst file when I removed that. I'll put that back, thank you for catching it.

@author:
Fixed, sorry about that.