Repository: ansible/ansible
Pull request: ansible-galaxy - add download option
##### SUMMARY
Add a `ansible-galaxy collection download` command that will download collections and it's dependencies to a format that is easy to install on an offline host. An example of this in action is the following

```
(ansible-py37) jborean:~/dev/ansible_collections$ ansible-galaxy collection download jborean93.windows -s https://galaxy-dev.ansible.com
[WARNING]: You are running the development version of Ansible. You should only run Ansible from "devel"
if you are modifying the Ansible engine, or trying out features under development. This is a rapidly
changing source of code and can become unstable at any point.
Process install dependency map
Starting collection download process to '/home/jborean/dev/ansible_collections/collections'
Downloading collection 'jborean93.windows' to '/home/jborean/dev/ansible_collections/collections/jborean93-windows-2.1.12.tar.gz'
Downloading collection 'alikins.collection_reqs_test' to '/home/jborean/dev/ansible_collections/collections/alikins-collection_reqs_test-2.1113.55.tar.gz'
Downloading collection 'alikins.collection_ntp' to '/home/jborean/dev/ansible_collections/collections/alikins-collection_ntp-0.1.182.tar.gz'
Downloading collection 'alikins.collection_inspect' to '/home/jborean/dev/ansible_collections/collections/alikins-collection_inspect-0.0.38.tar.gz'
Writing requirements.yml file of downloaded collections to '/home/jborean/dev/ansible_collections/collections/requirements.yml'

(ansible-py37) jborean:~/dev/ansible_collections$ ls collections/
alikins-collection_inspect-0.0.38.tar.gz       jborean93-windows-2.1.12.tar.gz
alikins-collection_ntp-0.1.182.tar.gz          requirements.yml
alikins-collection_reqs_test-2.1113.55.tar.gz

(ansible-py37) jborean:~/dev/ansible_collections$ cat collections/requirements.yml 
collections:
- name: jborean93-windows-2.1.12.tar.gz
  version: 2.1.12
- name: alikins-collection_reqs_test-2.1113.55.tar.gz
  version: 2.1113.55
- name: alikins-collection_ntp-0.1.182.tar.gz
  version: 0.1.182
- name: alikins-collection_inspect-0.0.38.tar.gz
  version: 0.0.38

(ansible-py37) jborean:~/dev/ansible_collections$ cd collections/

(ansible-py37) jborean:~/dev/ansible_collections/collections$ ansible-galaxy collection install -r requirements.yml 
[WARNING]: You are running the development version of Ansible. You should only run Ansible from "devel"
if you are modifying the Ansible engine, or trying out features under development. This is a rapidly
changing source of code and can become unstable at any point.
Process install dependency map
Starting collection install process
Installing 'jborean93.windows:2.1.12' to '/home/jborean/.ansible/collections/ansible_collections/jborean93/windows'
Installing 'alikins.collection_reqs_test:2.1113.55' to '/home/jborean/.ansible/collections/ansible_collections/alikins/collection_reqs_test'
Installing 'alikins.collection_ntp:0.1.182' to '/home/jborean/.ansible/collections/ansible_collections/alikins/collection_ntp'
Installing 'alikins.collection_inspect:0.0.38' to '/home/jborean/.ansible/collections/ansible_collections/alikins/collection_inspect'
```

Obviously that downloaded `collections` folder would be copied to another host but this just shows how simple it can be to install collections with dependencies offline after downloading them.

TODO:
- [x] Add unit tests (integration tests would be reliant on an actual Galaxy host)
- [ ] See if we should change what path a `requirements.yml` file should be relative to, the pwd (current) or relative to the requirements.yml file

Fixes https://github.com/ansible/ansible/issues/66534

##### ISSUE TYPE
- Feature Pull Request

##### COMPONENT NAME
ansible-galaxy

Problem statement:
ansible-galaxy cli should have an option to keep the collection tarball when installing collection
<!--- Verify first that your feature was not already discussed on GitHub -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Describe the new feature/improvement briefly below -->
Use case is for customers on air-gapped customers who want to take the tarball using a flash drive inside their network.
Currently that is not possible because cli does not keep the tarball after installing the collection.

##### ISSUE TYPE
- Feature Idea

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below, use your best guess if unsure -->
ansible-galaxy cli

##### ADDITIONAL INFORMATION
<!--- Describe how the feature would be used, why it is needed and what it would solve -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml

```

<!--- HINT: You can also paste gist.github.com links for larger files -->

Hints from issue discussion:
Files identified in the description:
* [`lib/ansible/galaxy`](https://github.com/ansible/ansible/blob/devel/lib/ansible/galaxy)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
@sivel @thaumos - This is a blocking issue for a few partners, is there someone that can pick this up? Thinking this could just be an added flag or something simple...
FYI: The other "half" of this feature https://github.com/ansible/galaxy/issues/2198
To mirror what I said elsewhere, this falls outside of the scope from what the galaxy cli project team is going to work on for the 2.10 release. We have other targets in flight that haven't been met as of yet that need completed.

I have no problem with it being added, but the project team will likely be unable to work on this feature, or at least not as part of the planned scope and not in the short term. To clarify on timelines, adding such a flag would land in 2.10, and no release date for 2.10 has been confirmed, but it's not soon.

We would be open to someone submitting a PR that includes this, and it will be reviewed like any other submission.
I've started to work on something like this https://github.com/ansible/ansible/pull/67632. It's reliant on a few other bits and pieces I've got going on but this might be able to make it into ansible-galaxy if approved by others.

Commit message: Fix doc suggestions

Patch to review:
diff --git a/changelogs/fragments/galaxy-download.yaml b/changelogs/fragments/galaxy-download.yaml
new file mode 100644
index 00000000000000..2887c3d09e8738
--- /dev/null
+++ b/changelogs/fragments/galaxy-download.yaml
@@ -0,0 +1,2 @@
+minor_changes:
+- ansible-galaxy - Add ``download`` option for ``ansible-galaxy collection`` to download collections and their dependencies for an offline install
diff --git a/docs/docsite/rst/user_guide/collections_using.rst b/docs/docsite/rst/user_guide/collections_using.rst
index 1d0632b34073ff..1ea1b1acf3798c 100644
--- a/docs/docsite/rst/user_guide/collections_using.rst
+++ b/docs/docsite/rst/user_guide/collections_using.rst
@@ -55,6 +55,60 @@ Configuring the ``ansible-galaxy`` client
 
 .. include:: ../shared_snippets/galaxy_server_list.txt
 
+.. _collections_downloading:
+
+Downloading collections
+=======================
+
+To download a collection and its dependencies for an offline install, run ``ansible-galaxy collection download``. This
+downloads the collections specified and their dependencies to the specified folder and creates a ``requirements.yml``
+file which can be used to install those collections on a host without access to a Galaxy server. All the collections
+are downloaded by default to the ``./collections`` folder.
+
+Just like the ``install`` command, the collections are sourced based on the
+:ref:`configured galaxy server config <galaxy_server_config>`. Even if a collection to download was specified by a URL
+or path to a tarball, the collection will be redownloaded from the configured Galaxy server.
+
+Collections can be specified as one or multiple collections or with a ``requirements.yml`` file just like
+``ansible-galaxy collection install``.
+
+To download a single collection and its dependencies:
+
+.. code-block:: bash
+
+   ansible-galaxy collection download my_namespace.my_collection
+
+To download a single collection at a specific version:
+
+.. code-block:: bash
+
+   ansible-galaxy collection download my_namespace.my_collection:1.0.0
+
+To download multiple collections either specify multiple collections as command line arguments as shown above or use a
+requirements file in the format documented with :ref:`collection_requirements_file`.
+
+.. code-block:: bash
+
+   ansible-galaxy collection download -r requirements.yml
+
+All the collections are downloaded by default to the ``./collections`` folder but you can use ``-p`` or
+``--download-path`` to specify another path:
+
+.. code-block:: bash
+
+   ansible-galaxy collection download my_namespace.my_collection -p ~/offline-collections
+
+Once you have downloaded the collections, the folder contains the collections specified, their dependencies, and a
+``requirements.yml`` file. You can use this folder as is with ``ansible-galaxy collection install`` to install the
+collections on a host without access to a Galaxy or Automation Hub server.
+
+.. code-block:: bash
+
+   # This must be run from the folder that contains the offline collections and requirements.yml file downloaded
+   # by the internet-connected host
+   cd ~/offline-collections
+   ansible-galaxy collection install -r requirements.yml
+
 .. _collections_listing:
 
 Listing collections
diff --git a/lib/ansible/cli/galaxy.py b/lib/ansible/cli/galaxy.py
index 96422b22fac496..6423829d968515 100644
--- a/lib/ansible/cli/galaxy.py
+++ b/lib/ansible/cli/galaxy.py
@@ -25,6 +25,7 @@
 from ansible.galaxy.collection import (
     build_collection,
     CollectionRequirement,
+    download_collections,
     find_existing_collections,
     install_collections,
     publish_collection,
@@ -162,6 +163,7 @@ def init_parser(self):
         collection = type_parser.add_parser('collection', help='Manage an Ansible Galaxy collection.')
         collection_parser = collection.add_subparsers(metavar='COLLECTION_ACTION', dest='action')
         collection_parser.required = True
+        self.add_download_options(collection_parser, parents=[common])
         self.add_init_options(collection_parser, parents=[common, force])
         self.add_build_options(collection_parser, parents=[common, force])
         self.add_publish_options(collection_parser, parents=[common])
@@ -184,6 +186,23 @@ def init_parser(self):
         self.add_info_options(role_parser, parents=[common, roles_path, offline])
         self.add_install_options(role_parser, parents=[common, force, roles_path])
 
+    def add_download_options(self, parser, parents=None):
+        download_parser = parser.add_parser('download', parents=parents,
+                                            help='Download collections and their dependencies as a tarball for an '
+                                                 'offline install.')
+        download_parser.set_defaults(func=self.execute_download)
+
+        download_parser.add_argument('args', help='Collection(s)', metavar='collection', nargs='*')
+
+        download_parser.add_argument('-n', '--no-deps', dest='no_deps', action='store_true', default=False,
+                                     help="Don't download collection(s) listed as dependencies.")
+
+        download_parser.add_argument('-p', '--download-path', dest='download_path',
+                                     default='./collections',
+                                     help='The directory to download the collections to.')
+        download_parser.add_argument('-r', '--requirements-file', dest='requirements',
+                                     help='A file containing a list of collections to be downloaded.')
+
     def add_init_options(self, parser, parents=None):
         galaxy_type = 'collection' if parser.metavar == 'COLLECTION_ACTION' else 'role'
 
@@ -714,6 +733,27 @@ def execute_build(self):
             collection_path = GalaxyCLI._resolve_path(collection_path)
             build_collection(collection_path, output_path, force)
 
+    def execute_download(self):
+        collections = context.CLIARGS['args']
+        no_deps = context.CLIARGS['no_deps']
+        download_path = context.CLIARGS['download_path']
+        ignore_certs = context.CLIARGS['ignore_certs']
+
+        requirements_file = context.CLIARGS['requirements']
+        if requirements_file:
+            requirements_file = GalaxyCLI._resolve_path(requirements_file)
+
+        requirements = self._require_one_of_collections_requirements(collections, requirements_file)
+
+        download_path = GalaxyCLI._resolve_path(download_path)
+        b_download_path = to_bytes(download_path, errors='surrogate_or_strict')
+        if not os.path.exists(b_download_path):
+            os.makedirs(b_download_path)
+
+        download_collections(requirements, download_path, self.api_servers, (not ignore_certs), no_deps)
+
+        return 0
+
     def execute_init(self):
         """
         Creates the skeleton framework of a role or collection that complies with the Galaxy metadata format.
diff --git a/lib/ansible/galaxy/collection.py b/lib/ansible/galaxy/collection.py
index f0077384873bb1..f6fa222486ba99 100644
--- a/lib/ansible/galaxy/collection.py
+++ b/lib/ansible/galaxy/collection.py
@@ -177,6 +177,17 @@ def add_requirement(self, parent, requirement):
 
         self.versions = new_versions
 
+    def download(self, b_path):
+        download_url = self._metadata.download_url
+        artifact_hash = self._metadata.artifact_sha256
+        headers = {}
+        self.api._add_auth_token(headers, download_url, required=False)
+
+        b_collection_path = _download_file(download_url, b_path, artifact_hash, self.api.validate_certs,
+                                           headers=headers)
+
+        return to_text(b_collection_path, errors='surrogate_or_strict')
+
     def install(self, path, b_temp_path):
         if self.skip:
             display.display("Skipping '%s' as it is already installed" % to_text(self))
@@ -188,13 +199,7 @@ def install(self, path, b_temp_path):
         display.display("Installing '%s:%s' to '%s'" % (to_text(self), self.latest_version, collection_path))
 
         if self.b_path is None:
-            download_url = self._metadata.download_url
-            artifact_hash = self._metadata.artifact_sha256
-            headers = {}
-            self.api._add_auth_token(headers, download_url, required=False)
-
-            self.b_path = _download_file(download_url, b_temp_path, artifact_hash, self.api.validate_certs,
-                                         headers=headers)
+            self.b_path = self.download(b_temp_path)
 
         if os.path.exists(b_collection_path):
             shutil.rmtree(b_collection_path)
@@ -492,6 +497,41 @@ def build_collection(collection_path, output_path, force):
     _build_collection_tar(b_collection_path, b_collection_output, collection_manifest, file_manifest)
 
 
+def download_collections(collections, output_path, apis, validate_certs, no_deps):
+    """
+    Download Ansible collections as their tarball from a Galaxy server to the path specified and creates a requirements
+    file of the downloaded requirements to be used for an install.
+
+    :param collections: The collections to download, should be a list of tuples with (name, requirement, Galaxy Server).
+    :param output_path: The path to download the collections to.
+    :param apis: A list of GalaxyAPIs to query when search for a collection.
+    :param validate_certs: Whether to validate the certificate if downloading a tarball from a non-Galaxy host.
+    :param no_deps: Ignore any collection dependencies and only download the base requirements.
+    """
+    with _tempdir() as b_temp_path:
+        display.display("Process install dependency map")
+        with _display_progress():
+            dep_map = _build_dependency_map(collections, [], b_temp_path, apis, validate_certs, True, True, no_deps)
+
+        requirements = []
+        display.display("Starting collection download process to '%s'" % output_path)
+        with _display_progress():
+            for name, requirement in dep_map.items():
+                collection_filename = "%s-%s-%s.tar.gz" % (requirement.namespace, requirement.name,
+                                                           requirement.latest_version)
+                dest_path = os.path.join(output_path, collection_filename)
+                requirements.append({'name': collection_filename, 'version': requirement.latest_version})
+
+                display.display("Downloading collection '%s' to '%s'" % (name, dest_path))
+                b_temp_download_path = requirement.download(b_temp_path)
+                shutil.move(b_temp_download_path, to_bytes(dest_path, errors='surrogate_or_strict'))
+
+            requirements_path = os.path.join(output_path, 'requirements.yml')
+            display.display("Writing requirements.yml file of downloaded collections to '%s'" % requirements_path)
+            with open(to_bytes(requirements_path, errors='surrogate_or_strict'), mode='wb') as req_fd:
+                req_fd.write(to_bytes(yaml.safe_dump({'collections': requirements}), errors='surrogate_or_strict'))
+
+
 def publish_collection(collection_path, api, wait, timeout):
     """
     Publish an Ansible collection tarball into an Ansible Galaxy server.
diff --git a/test/integration/targets/ansible-galaxy-collection/tasks/download.yml b/test/integration/targets/ansible-galaxy-collection/tasks/download.yml
new file mode 100644
index 00000000000000..f5bfe69d194af1
--- /dev/null
+++ b/test/integration/targets/ansible-galaxy-collection/tasks/download.yml
@@ -0,0 +1,88 @@
+---
+- name: create test download dir
+  file:
+    path: '{{ galaxy_dir }}/download'
+    state: directory
+
+- name: download collection with multiple dependencides
+  command: ansible-galaxy collection download parent_dep.parent_collection -s {{ fallaxy_galaxy_server }}
+  register: download_collection
+  args:
+    chdir: '{{ galaxy_dir }}/download'
+
+- name: get result of download collection with multiple dependencies
+  find:
+    path: '{{ galaxy_dir }}/download/collections'
+    file_type: file
+  register: download_collection_actual
+
+- name: assert download collection with multiple dependencies
+  assert:
+    that:
+    - '"Downloading collection ''parent_dep.parent_collection'' to" in download_collection.stdout'
+    - '"Downloading collection ''child_dep.child_collection'' to" in download_collection.stdout'
+    - '"Downloading collection ''child_dep.child_dep2'' to" in download_collection.stdout'
+    - download_collection_actual.examined == 4
+    - download_collection_actual.matched == 4
+    - (download_collection_actual.files[0].path | basename) in ['requirements.yml', 'child_dep-child_dep2-1.2.2.tar.gz', 'child_dep-child_collection-0.9.9.tar.gz', 'parent_dep-parent_collection-1.0.0.tar.gz']
+    - (download_collection_actual.files[1].path | basename) in ['requirements.yml', 'child_dep-child_dep2-1.2.2.tar.gz', 'child_dep-child_collection-0.9.9.tar.gz', 'parent_dep-parent_collection-1.0.0.tar.gz']
+    - (download_collection_actual.files[2].path | basename) in ['requirements.yml', 'child_dep-child_dep2-1.2.2.tar.gz', 'child_dep-child_collection-0.9.9.tar.gz', 'parent_dep-parent_collection-1.0.0.tar.gz']
+    - (download_collection_actual.files[3].path | basename) in ['requirements.yml', 'child_dep-child_dep2-1.2.2.tar.gz', 'child_dep-child_collection-0.9.9.tar.gz', 'parent_dep-parent_collection-1.0.0.tar.gz']
+
+- name: test install of download requirements file
+  command: ansible-galaxy collection install -r requirements.yml -p '{{ galaxy_dir }}/download'
+  args:
+    chdir: '{{ galaxy_dir }}/download/collections'
+  register: install_download
+
+- name: get result of test install of download requirements file
+  slurp:
+    path: '{{ galaxy_dir }}/download/ansible_collections/{{ collection.namespace }}/{{ collection.name }}/MANIFEST.json'
+  register: install_download_actual
+  loop_control:
+    loop_var: collection
+  loop:
+  - namespace: parent_dep
+    name: parent_collection
+  - namespace: child_dep
+    name: child_collection
+  - namespace: child_dep
+    name: child_dep2
+
+- name: assert test install of download requirements file
+  assert:
+    that:
+    - '"Installing ''parent_dep.parent_collection:1.0.0'' to" in install_download.stdout'
+    - '"Installing ''child_dep.child_collection:0.9.9'' to" in install_download.stdout'
+    - '"Installing ''child_dep.child_dep2:1.2.2'' to" in install_download.stdout'
+    - (install_download_actual.results[0].content | b64decode | from_json).collection_info.version == '1.0.0'
+    - (install_download_actual.results[1].content | b64decode | from_json).collection_info.version == '0.9.9'
+    - (install_download_actual.results[2].content | b64decode | from_json).collection_info.version == '1.2.2'
+
+- name: create test requirements file for download
+  copy:
+    content: |
+      collections:
+      - name: namespace1.name1
+        version: 1.1.0-beta.1
+
+    dest: '{{ galaxy_dir }}/download/download.yml'
+
+- name: download collection with req to custom dir
+  command: ansible-galaxy collection download -r '{{ galaxy_dir }}/download/download.yml' -s {{ fallaxy_ah_server }} -p '{{ galaxy_dir }}/download/collections-custom'
+  register: download_req_custom_path
+
+- name: get result of download collection with req to custom dir
+  find:
+    path: '{{ galaxy_dir }}/download/collections-custom'
+    file_type: file
+  register: download_req_custom_path_actual
+
+- name: assert download collection with multiple dependencies
+  assert:
+    that:
+    - '"Downloading collection ''namespace1.name1'' to" in download_req_custom_path.stdout'
+    - download_req_custom_path_actual.examined == 2
+    - download_req_custom_path_actual.matched == 2
+    - (download_req_custom_path_actual.files[0].path | basename) in ['requirements.yml', 'namespace1-name1-1.1.0-beta.1.tar.gz']
+    - (download_req_custom_path_actual.files[1].path | basename) in ['requirements.yml', 'namespace1-name1-1.1.0-beta.1.tar.gz']
diff --git a/test/integration/targets/ansible-galaxy-collection/tasks/main.yml b/test/integration/targets/ansible-galaxy-collection/tasks/main.yml
index 6dc2ab1c01a8b9..0a34acc65eedd4 100644
--- a/test/integration/targets/ansible-galaxy-collection/tasks/main.yml
+++ b/test/integration/targets/ansible-galaxy-collection/tasks/main.yml
@@ -31,7 +31,7 @@
     server: '{{ fallaxy_ah_server }}'
 
 # We use a module for this so we can speed up the test time.
-- name: setup test collections for install test
+- name: setup test collections for install and download test
   setup_collections:
     server: '{{ fallaxy_galaxy_server }}'
     token: '{{ fallaxy_token }}'
@@ -148,3 +148,6 @@
     server: '{{ fallaxy_galaxy_server }}'
   - name: automation_hub
     server: '{{ fallaxy_ah_server }}'
+
+- name: run ansible-galaxy collection download tests
+  include_tasks: download.yml


Code review comments:
diff_hunk: @@ -492,6 +497,41 @@ def build_collection(collection_path, output_path, force):
     _build_collection_tar(b_collection_path, b_collection_output, collection_manifest, file_manifest)
 
 
+def download_collections(collections, output_path, apis, validate_certs, no_deps):
+    """
+    Download Ansible collections as their tarball from a Galaxy server to the path specified and creates a requirements
+    file of the downloaded requirements to be used for an install.
+
+    :param collections: The collections to download, should be a list of tuples with (name, requirement, Galaxy Server).
+    :param output_path: The path to download the collections to.
+    :param apis: A list of GalaxyAPIs to query when search for a collection.
+    :param validate_certs: Whether to validate the certificate if downloading a tarball from a non-Galaxy host.
+    :param no_deps: Ignore any collection dependencies and only download the base requirements.
+    """
+    with _tempdir() as b_temp_path:
+        display.display("Process install dependency map")
+        with _display_progress():
+            dep_map = _build_dependency_map(collections, [], b_temp_path, apis, validate_certs, True, True, no_deps)
path: lib/ansible/galaxy/collection.py
text: @user1:
Should `download` take the `--pre` flag so this can keep parity with `ansible-galaxy collection install`?

@user1:
freeipa.ansible_freeipa is an example of a published pre-release-only collection

@author:
Yea that is a good idea, I just did a basic rebase after sivel's changes but I'll add that flag now.

@author:
Have added the flag and tested that collection

```
ansible-galaxy collection download freeipa.ansible_freeipa --pre
Process install dependency map
Starting collection download process to '/home/jborean/Documents/collections'
Downloading collection 'freeipa.ansible_freeipa' to '/home/jborean/Documents/collections/freeipa-ansible_freeipa-0.1.8-1.tar.gz'
Writing requirements.yml file of downloaded collections to '/home/jborean/Documents/collections/requirements.yml'
```