Repository: ansible/ansible
Pull request: ansible-galaxy - fall back to galaxy.yml when listing collections
##### SUMMARY
Fixes #67490
Supersedes #67630

Use galaxy.yml as a fallback for `ansible-galaxy collection list` if the collection doesn't contain a MANIFEST.json file.

##### ISSUE TYPE
- Bugfix Pull Request

Problem statement:
ansible-galaxy collection list should examine galaxy.yml for information
##### SUMMARY

The list subcommand throws a lot of warnings about a missing version if the collection was not built and installed via an artifact. This information can often be obtained in the galaxy.yml though, so it would make sense to fallback to that ...

```
(ansible_base) [vagrant@centos8 ~]$ ansible-galaxy collection list
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/azure/azcollection' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cyberark/bizdev' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/google/cloud' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/netbox_community/ansible_modules' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/splunk/enterprise_security' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/ansible/netcommon' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/check_point/mgmt' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/f5networks/f5_modules' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/ibm/qradar' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/openstack/cloud' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/vyos/vyos' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/arista/eos' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/iosxr' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/ucs' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/aci' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/meraki' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/intersight' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/mso' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/ios' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/cisco/nxos' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/fortinet/fortios' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/junipernetworks/junos' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/purestorage/flasharray' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/purestorage/flashblade' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/awx/awx' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/community/grafana' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/community/kubernetes' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/community/amazon' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/community/vmware' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/community/general' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/gavinfish/azuretest' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/netapp/aws' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/netapp/elementsw' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/netapp/ontap' does not have a MANIFEST.json file, cannot detect version.
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/servicenow/servicenow' does not have a MANIFEST.json file, cannot detect version.

# /home/vagrant/.ansible/collections/ansible_collections
Collection                       Version
-------------------------------- -------
ansible.netcommon                *      
arista.eos                       *      
awx.awx                          *      
azure.azcollection               *      
check_point.mgmt                 *      
cisco.aci                        *      
cisco.intersight                 *      
cisco.ios                        *      
cisco.iosxr                      *      
cisco.meraki                     *      
cisco.mso                        *      
cisco.nxos                       *      
cisco.ucs                        *      
community.amazon                 *      
community.general                *      
community.grafana                *      
community.kubernetes             *      
community.vmware                 *      
cyberark.bizdev                  *      
f5networks.f5_modules            *      
fortinet.fortios                 *      
gavinfish.azuretest              *      
google.cloud                     *      
ibm.qradar                       *      
junipernetworks.junos            *      
netapp.aws                       *      
netapp.elementsw                 *      
netapp.ontap                     *      
netbox_community.ansible_modules *      
openstack.cloud                  *      
purestorage.flasharray           *      
purestorage.flashblade           *      
servicenow.servicenow            *      
splunk.enterprise_security       *      
vyos.vyos                        *
```


##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
ansible-galaxy collection list

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
devel
```

Hints from issue discussion:
We could also extend this logic to verify ...

```
(ansible_base) [vagrant@centos8 ~]$ ansible-galaxy collection verify -s http://sivel.eng.ansible.com/api openstack.cloud
[WARNING]: Collection at '/home/vagrant/.ansible/collections/ansible_collections/openstack/cloud' does not have a MANIFEST.json file, cannot detect version.
openstack.cloud has the version '*' but is being compared to '1.0.0'
```
@jctanner I'm not sure if it should work, for verify at least. All built and then published collections should have a MANIFEST (they are built from the galaxy.yml), so it only shouldn't exist after installation if it has been actively deleted or if the collection wasn't built with `ansible-galaxy collection build` and something janky is happening instead. (Also note that galaxy.yml doesn't exist after installation - and since verify is verifying installations...)

But I think the error should be fixed for `verify` since it's misleading.
Fixing the error message for `verify` in https://github.com/ansible/ansible/pull/67498.
This makes sense for `collection list`. We can just fall back to inspecting the local `galaxy.yml` file favoring `MANIFEST.json`. There's already a method to ingest that file, so I can use that.

Commit message: add a note to the documentation

Patch to review:
diff --git a/docs/docsite/rst/user_guide/collections_using.rst b/docs/docsite/rst/user_guide/collections_using.rst
index 1ea1b1acf3798c..230c6407dd1f92 100644
--- a/docs/docsite/rst/user_guide/collections_using.rst
+++ b/docs/docsite/rst/user_guide/collections_using.rst
@@ -114,7 +114,7 @@ collections on a host without access to a Galaxy or Automation Hub server.
 Listing collections
 ===================
 
-To list installed collections, run ``ansible-galaxy collection list``. This shows all of the installed collections found in the configured collections search paths. The path where the collections are located are displayed as well as version information. If no version information is available, a ``*`` is displayed for the version number.
+To list installed collections, run ``ansible-galaxy collection list``. This shows all of the installed collections found in the configured collections search paths. It will also show collections under development which contain a galaxy.yml file instead of a MANIFEST.json. The path where the collections are located are displayed as well as version information. If no version information is available, a ``*`` is displayed for the version number.
 
 .. code-block:: shell
 
diff --git a/lib/ansible/cli/galaxy.py b/lib/ansible/cli/galaxy.py
index 89a9dcc547d10f..11bd38f13f04ba 100644
--- a/lib/ansible/cli/galaxy.py
+++ b/lib/ansible/cli/galaxy.py
@@ -1221,7 +1221,7 @@ def execute_list_collection(self):
                     continue
 
                 collection_found = True
-                collection = CollectionRequirement.from_path(b_collection_path, False)
+                collection = CollectionRequirement.from_path(b_collection_path, False, fallback_metadata=True)
                 fqcn_width, version_width = _get_collection_widths(collection)
 
                 _display_header(collection_path, 'Collection', 'Version', fqcn_width, version_width)
@@ -1232,7 +1232,7 @@ def execute_list_collection(self):
                 collection_path = validate_collection_path(path)
                 if os.path.isdir(collection_path):
                     display.vvv("Searching {0} for collections".format(collection_path))
-                    collections = find_existing_collections(collection_path)
+                    collections = find_existing_collections(collection_path, fallback_metadata=True)
                 else:
                     # There was no 'ansible_collections/' directory in the path, so there
                     # or no collections here.
diff --git a/lib/ansible/galaxy/collection.py b/lib/ansible/galaxy/collection.py
index 51d51b1cdfccd6..7b3fed434d39e4 100644
--- a/lib/ansible/galaxy/collection.py
+++ b/lib/ansible/galaxy/collection.py
@@ -224,7 +224,7 @@ def install(self, path, b_temp_path):
                         _extract_tar_file(collection_tar, file_name, b_collection_path, b_temp_path,
                                           expected_hash=file_info['chksum_sha256'])
                     else:
-                        os.makedirs(os.path.join(b_collection_path, to_bytes(file_name, errors='surrogate_or_strict')))
+                        os.makedirs(os.path.join(b_collection_path, to_bytes(file_name, errors='surrogate_or_strict')), mode=0o0755)
         except Exception:
             # Ensure we don't leave the dir behind in case of a failure.
             shutil.rmtree(b_collection_path)
@@ -386,7 +386,7 @@ def from_tar(b_path, force, parent=None):
                                      metadata=meta, files=files, allow_pre_releases=allow_pre_release)
 
     @staticmethod
-    def from_path(b_path, force, parent=None):
+    def from_path(b_path, force, parent=None, fallback_metadata=False):
         info = {}
         for b_file_name, property_name in CollectionRequirement._FILE_MAPPING:
             b_file_path = os.path.join(b_path, b_file_name)
@@ -399,6 +399,13 @@ def from_path(b_path, force, parent=None):
                 except ValueError:
                     raise AnsibleError("Collection file at '%s' does not contain a valid json string."
                                        % to_native(b_file_path))
+        if not info and fallback_metadata:
+            b_galaxy_path = os.path.join(b_path, b'galaxy.yml')
+            if os.path.exists(b_galaxy_path):
+                collection_meta = _get_galaxy_yml(b_galaxy_path)
+                info['files_file'] = _build_files_manifest(b_path, collection_meta['namespace'], collection_meta['name'],
+                                                           collection_meta['build_ignore'])
+                info['manifest_file'] = _build_manifest(**collection_meta)
 
         allow_pre_release = False
         if 'manifest_file' in info:
@@ -419,8 +426,11 @@ def from_path(b_path, force, parent=None):
 
             dependencies = manifest['dependencies']
         else:
-            display.warning("Collection at '%s' does not have a MANIFEST.json file, cannot detect version."
-                            % to_text(b_path))
+            if fallback_metadata:
+                warning = "Collection at '%s' does not have a galaxy.yml or a MANIFEST.json file, cannot detect version."
+            else:
+                warning = "Collection at '%s' does not have a MANIFEST.json file, cannot detect version."
+            display.warning(warning % to_text(b_path))
             parent_dir, name = os.path.split(to_text(b_path, errors='surrogate_or_strict'))
             namespace = os.path.split(parent_dir)[1]
 
@@ -992,7 +1002,7 @@ def reset_stat(tarinfo):
         display.display('Created collection for %s at %s' % (collection_name, to_text(b_tar_path)))
 
 
-def find_existing_collections(path):
+def find_existing_collections(path, fallback_metadata=False):
     collections = []
 
     b_path = to_bytes(path, errors='surrogate_or_strict')
@@ -1004,7 +1014,7 @@ def find_existing_collections(path):
         for b_collection in os.listdir(b_namespace_path):
             b_collection_path = os.path.join(b_namespace_path, b_collection)
             if os.path.isdir(b_collection_path):
-                req = CollectionRequirement.from_path(b_collection_path, False)
+                req = CollectionRequirement.from_path(b_collection_path, False, fallback_metadata=fallback_metadata)
                 display.vvv("Found installed collection %s:%s at '%s'" % (to_text(req), req.latest_version,
                                                                           to_text(b_collection_path)))
                 collections.append(req)
diff --git a/test/integration/targets/ansible-galaxy/runme.sh b/test/integration/targets/ansible-galaxy/runme.sh
index 3409352b8b7aa6..22f773a97f4db5 100755
--- a/test/integration/targets/ansible-galaxy/runme.sh
+++ b/test/integration/targets/ansible-galaxy/runme.sh
@@ -179,6 +179,10 @@ sed -i -e 's#^version:.*#version: 2.5.0#' ansible_test/zoo/galaxy.yml
 ansible-galaxy collection build ansible_test/zoo
 ansible-galaxy collection install ansible_test-zoo-2.5.0.tar.gz -p ./local
 
+# Test listing a collection that contains a galaxy.yml
+ansible-galaxy collection init "ansible_test.development"
+mv ./ansible_test/development "${galaxy_testdir}/local/ansible_collections/ansible_test/"
+
 export ANSIBLE_COLLECTIONS_PATHS=~/.ansible/collections:${galaxy_testdir}/local
 
 f_ansible_galaxy_status \
@@ -186,7 +190,7 @@ f_ansible_galaxy_status \
 
     ansible-galaxy collection list -p ./install | tee out.txt
 
-    [[ $(grep -c ansible_test out.txt) -eq 4 ]]
+    [[ $(grep -c ansible_test out.txt) -eq 5 ]]
 
 f_ansible_galaxy_status \
     "collection list specific collection"
@@ -195,6 +199,13 @@ f_ansible_galaxy_status \
 
     [[ $(grep -c 'ansible_test\.airport' out.txt) -eq 1 ]]
 
+f_ansible_galaxy_status \
+    "collection list specific collection which contains galaxy.yml"
+
+    ansible-galaxy collection list -p ./install ansible_test.development | tee out.txt
+
+    [[ $(grep -c 'ansible_test\.development' out.txt) -eq 1 ]]
+
 f_ansible_galaxy_status \
     "collection list specific collection found in multiple places"
 


Code review comments:
diff_hunk: @@ -195,6 +199,13 @@ f_ansible_galaxy_status \
 
     [[ $(grep -c 'ansible_test\.airport' out.txt) -eq 1 ]]
 
+f_ansible_galaxy_status \
+    "collection list specific collection which contains galaxy.yml"
+
+    ansible-galaxy collection list -p ./install ansible_test.development | tee out.txt
+
+    [[ $(grep -c 'ansible_test\.development' out.txt) -eq 1 ]]
+
path: test/integration/targets/ansible-galaxy/runme.sh
text: @user1:
This test won't catch the warning that shouldn't be there.
```suggestion
    ansible-galaxy collection list -p ./install ansible_test.development 2>&1 | tee out.txt

    [[ $(grep -c 'ansible_test\.development' out.txt) -eq 1 ]]
    [[ $(grep -c 'WARNING' out.txt) -eq 0 ]]

```