Repository: ansible/ansible

Issue:
Ansible Galaxy: Keep clone of repositories
##### ISSUE TYPE
Feature Idea

##### COMPONENT NAME
core

##### ANSIBLE VERSION
2.1

##### CONFIGURATION


##### OS / ENVIRONMENT


##### SUMMARY
I'd appreciate if I could use ansible-galaxy to download the git repositories and keep them.
Right now ansible-galaxy does, e.g. git archive to get the contents and then discards the cloned git repository.
I'd like to keep the git repository, because I can then track changes more easily.


##### STEPS TO REPRODUCE


##### EXPECTED RESULTS


##### ACTUAL RESULTS





Discussion:
This might be best implemented as an option in the .yml file (e.g. develop: true) to clone and fetch/(merge/warn) on particular roles.

It looks like there was an attempt to do this already.  The PR is still pending, and seemed to be waiting for v2.

https://github.com/ansible/ansible/pull/6537

PR was closed, this feature has not been implemented.

> PR was closed, this feature has not been implemented.

Just to elaborate, this was not blocked or rejected, rather because of time/development/release issues. So working on a PR for this feature might still be worthwhile.

@muelli Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information. 

Here are the items we could not find in your description:
- issue type
- ansible version
- component name

Please set the description of this issue with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE.md

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: issue_missing_data --->
Just to be clear, is the ask to use a flag or option within ansible-galaxy to do a git clone as opposed to  just downloading a tarball?
This should be a flag for the ```ansible-galaxy``` command. Opposed to the view of @umeboshi2 I think it is best not to touch ```requirements.yml``` for this use-case. Only maintainers - tool smiths - will need this flag for integration testing and developing roles in separate repositories.
I would have to disagree with @bbaassssiiee. while you are correct that this option would primarily be used by maintainers/tool smiths, Many of them will likely (and hopefully do) have automation in place as part of their validation flow. using tools like Ansible Tower/AWX which supports the requirements.yml convention, but doesn't allow you to modify the command that is executed to download the roles. 

As a result, an option in the requirements.yml file would be the best choice to provide this feature.

Additionally, for public repos, it could aid in the collaborative development of roles, allowing for easier PR back to the master branch.
Agree with @bbaassssiiee Automation flows should exist, but I don't think developers should be coding on roles checked out by tower.

- checkout all roles for a play to a dev host somewhere (ansible-galaxy install --clone)
- update roles and check with some local testing
- commit and push so tower/jenkins can do more rigorous testing.



Resolving patch:
diff --git a/lib/ansible/cli/galaxy.py b/lib/ansible/cli/galaxy.py
--- a/lib/ansible/cli/galaxy.py
+++ b/lib/ansible/cli/galaxy.py
@@ -91,6 +91,8 @@ def set_action(self):
                                    help='Ignore errors and continue with the next specified role.')
             self.parser.add_option('-n', '--no-deps', dest='no_deps', action='store_true', default=False, help='Don\'t download roles listed as dependencies')
             self.parser.add_option('-r', '--role-file', dest='role_file', help='A file containing a list of roles to be imported')
+            self.parser.add_option('-g', '--keep-scm-meta', dest='keep_scm_meta', action='store_true',
+                                   default=False, help='Use tar instead of the scm archive option when packaging the role')
         elif self.action == "remove":
             self.parser.set_usage("usage: %prog remove role1 role2 ...")
         elif self.action == "list":
diff --git a/lib/ansible/galaxy/role.py b/lib/ansible/galaxy/role.py
--- a/lib/ansible/galaxy/role.py
+++ b/lib/ansible/galaxy/role.py
@@ -194,17 +194,12 @@ def fetch(self, role_data):
         return False
 
     def install(self):
-        # the file is a tar, so open it that way and extract it
-        # to the specified (or default) roles directory
-        local_file = False
 
         if self.scm:
             # create tar file from scm url
-            tmp_file = RoleRequirement.scm_archive_role(**self.spec)
+            tmp_file = RoleRequirement.scm_archive_role(keep_scm_meta=self.options.keep_scm_meta, **self.spec)
         elif self.src:
             if os.path.isfile(self.src):
-                # installing a local tar.gz
-                local_file = True
                 tmp_file = self.src
             elif '://' in self.src:
                 role_data = self.src
@@ -337,7 +332,7 @@ def install(self):
 
                 # return the parsed yaml metadata
                 display.display("- %s was installed successfully" % str(self))
-                if not local_file:
+                if not (self.src and os.path.isfile(self.src)):
                     try:
                         os.unlink(tmp_file)
                     except (OSError, IOError) as e:
diff --git a/lib/ansible/playbook/role/requirement.py b/lib/ansible/playbook/role/requirement.py
--- a/lib/ansible/playbook/role/requirement.py
+++ b/lib/ansible/playbook/role/requirement.py
@@ -23,6 +23,7 @@
 import shutil
 import subprocess
 import tempfile
+import tarfile
 
 from ansible.errors import AnsibleError
 from ansible.module_utils.six import string_types
@@ -182,7 +183,7 @@ def role_yaml_parse(role):
         return role
 
     @staticmethod
-    def scm_archive_role(src, scm='git', name=None, version='HEAD'):
+    def scm_archive_role(src, scm='git', name=None, version='HEAD', keep_scm_meta=False):
         if scm not in ['hg', 'git']:
             raise AnsibleError("- scm %s is not currently supported" % scm)
         tempdir = tempfile.mkdtemp()
@@ -208,24 +209,31 @@ def scm_archive_role(src, scm='git', name=None, version='HEAD'):
                 raise AnsibleError("- command %s failed in directory %s (rc=%s)" % (' '.join(checkout_cmd), tempdir, rc))
 
         temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.tar')
-        if scm == 'hg':
+        archive_cmd = None
+        if keep_scm_meta:
+            display.vvv('tarring %s from %s to %s' % (name, tempdir, temp_file.name))
+            with tarfile.open(temp_file.name, "w") as tar:
+                tar.add(os.path.join(tempdir, name), arcname=name)
+        elif scm == 'hg':
             archive_cmd = ['hg', 'archive', '--prefix', "%s/" % name]
             if version:
                 archive_cmd.extend(['-r', version])
             archive_cmd.append(temp_file.name)
-        if scm == 'git':
+        elif scm == 'git':
             archive_cmd = ['git', 'archive', '--prefix=%s/' % name, '--output=%s' % temp_file.name]
             if version:
                 archive_cmd.append(version)
             else:
                 archive_cmd.append('HEAD')
 
-        with open('/dev/null', 'w') as devnull:
-            popen = subprocess.Popen(archive_cmd, cwd=os.path.join(tempdir, name),
-                                     stderr=devnull, stdout=devnull)
-            rc = popen.wait()
-        if rc != 0:
-            raise AnsibleError("- command %s failed in directory %s (rc=%s)" % (' '.join(archive_cmd), tempdir, rc))
+        if archive_cmd is not None:
+            display.vvv('archiving %s' % archive_cmd)
+            with open('/dev/null', 'w') as devnull:
+                popen = subprocess.Popen(archive_cmd, cwd=os.path.join(tempdir, name),
+                                         stderr=devnull, stdout=devnull)
+                rc = popen.wait()
+            if rc != 0:
+                raise AnsibleError("- command %s failed in directory %s (rc=%s)" % (' '.join(archive_cmd), tempdir, rc))
 
         shutil.rmtree(tempdir, ignore_errors=True)
         return temp_file.name


Test patch: