Repository: ansible/ansible-lint

Issue:
[405] Remote package tasks should have a retry - too nitpicky?
# Issue Type

- Feature request

# Ansible and Ansible Lint details

```
$ ansible --version
ansible 2.7.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/jgeerling/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15 (default, Jul 17 2018, 19:41:20) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]

$ ansible-lint --version
ansible-lint 4.0.0
```

- ansible installation method: pip
- ansible-lint installation method: pip

# Desired Behaviour

Using package modules (`apt`, `yum`, `dnf`, `package`, etc.) should not require the user to always add 3+ extra lines of configuration per use to resolve 405.

Rather, if it's truly important enough that we add retries that it should be a lint warning... maybe it should be the responsibility of ansible itself, and it could be a module parameter?

So I guess this is more of a 'question request'—since 405 hits pretty much every playbook, role, and tiny ansible script I've ever written, and would require me adding thousands of lines of the exact same thing to all those hundreds of instances:

```yaml
register: task_result
until: task_result is success
retries: 10
delay: 2
```

...might it not be better to make this the responsibility of Ansible itself? Otherwise, this is one of those warnings I think many might just drop. Seems extremely nitpicky to me (even when rated as a `LOW`).

# Actual Behaviour (Bug report only)

The task:

```
- name: Update apt cache (Debian).
  apt: update_cache=true cache_valid_time=86400
  when: ansible_os_family == 'Debian'
```

(or any task with apt, yum, dnf, package, etc.)

results in:

```
[405] Remote package tasks should have a retry
/Users/jgeerling/Dropbox/VMs/roles/geerlingguy.git/tasks/main.yml:2
Task/Handler: Ensure git is installed (RedHat).
```


Discussion:
It also is a little more annoying to me, probably, because most of my roles support 4+ OSes, so many of them have 5-10 or more package-related tasks, and that racks up a _lot_ of notices!

For example, my MySQL role:

```
Starting import: task_id=348865, repository=geerlingguy/ansible-role-mysql

===== LINTING ROLE =====
yamllint OK.
ansible-lint Warnings:
./galaxy/downloads/tmpaWPbQE/tasks/setup-Archlinux.yml:2: [E405] Remote package tasks should have a retry
./galaxy/downloads/tmpaWPbQE/tasks/setup-Archlinux.yml:5: [E405] Remote package tasks should have a retry
./galaxy/downloads/tmpaWPbQE/tasks/setup-Debian.yml:6: [E405] Remote package tasks should have a retry
./galaxy/downloads/tmpaWPbQE/tasks/setup-Debian.yml:14: [E405] Remote package tasks should have a retry
./galaxy/downloads/tmpaWPbQE/tasks/setup-Debian.yml:17: [E405] Remote package tasks should have a retry
./galaxy/downloads/tmpaWPbQE/tasks/setup-RedHat.yml:2: [E405] Remote package tasks should have a retry
./galaxy/downloads/tmpaWPbQE/tasks/setup-RedHat.yml:9: [E405] Remote package tasks should have a retry
```

And it's not nearly as complex as some others I maintain.
Just to +1 this and add another possible concern...
yum and dnf both seem to retry 10 times by default:

> retries Set the number of times any attempt to retrieve a file should retry before returning an error. Setting this to '0' makes yum try forever. Default is '10'.
https://linux.die.net/man/5/yum.conf
https://dnf.readthedocs.io/en/latest/conf_ref.html

You can check my math but if I start putting:
```yaml
register: task_result
until: task_result is success
retries: 10
delay: 2
```
on all my yum/dnf tasks am I not actually setting up 100 retries? This is fairly misleading to the end user who likely thinks they are getting something reasonable like 10 but are in fact getting each ansible retry multiplied by the package manager's default.
apt has a similar 'retries' setting that could possibly be in play depending on how the user has their apt.conf setup.

Given that different package managers are already aware of the _'Package operations are unreliable'_ issue and are in some cases already handling it I'm not sure it makes sense to treat every package manager equally and request this be added across the board.
This looks like a topic for proposal https://github.com/ansible/proposals/issues/new
@webknjaz - Not sure about that—this is something that package managers sometimes handle on their own, sometimes don't, but as a general rule, I don't think we should try to enforce retries across the board on all the packaging-related modules.

As I put in the title, it seems too nitpicky. _Separately_, maybe retry logic could be added in Ansible by default for certain packaging-related modules _if those modules have a lot of flakiness_.

But in my experience, a failure of yum, apt, or dnf due to network issues is exceedingly rare... though most of my tests run in the US and EU :)
Agreed, it is nitpicky. Linting should enforce best practices. Retries are not required nor sometimes even wanted and as  but when Ansible itself doesn't enforce, warn or otherwise break from the lack of this rule. It is nice to have if that's what you're looking for an run into those issues, but maybe it should be instead tabled and added to a set of non-default rules that users can activate/use if they explicitly add the rule.
Looking at community feedback for this rule, we’ve had more comments with concerns than support. The concerns are all good points. A non-default rules folder (suggested here and historically in other issues) might be the best option, to move this rule from the default rules to “extra_rules”.
After more discussion we will drop rule 405 in the next release 4.1.0. If we find a greater need for a non-default rules folder, then that functionality can be discussed more and spec’d out.

Resolving patch:
diff --git a/lib/ansiblelint/rules/PackageHasRetryRule.py b/lib/ansiblelint/rules/PackageHasRetryRule.py
deleted file mode 100644
--- a/lib/ansiblelint/rules/PackageHasRetryRule.py
+++ /dev/null
@@ -1,134 +0,0 @@
-# Copyright (c) 2016, Will Thames and contributors
-# Copyright (c) 2018, Ansible Project
-
-from ansiblelint import AnsibleLintRule
-
-
-class PackageHasRetryRule(AnsibleLintRule):
-    id = '405'
-    shortdesc = 'Remote package tasks should have a retry'
-    description = (
-        'Package operations are unreliable as they require '
-        'network communication and the availability of remote '
-        'servers. To mitigate the potential problems, retries '
-        'should be used via '
-        '``register: my_result`` and ``until: my_result is succeeded``'
-    )
-    severity = 'LOW'
-    tags = ['module', 'reliability']
-    version_added = 'v4.0.0'
-
-    # module list generated with:
-    # find lib/ansible/modules/packaging/ -type f -printf '%f\n' \
-    #   | sort | awk -F '/' \
-    #   '/__|dpkg|_repo|_facts|_sub|_chan/{next} {split($NF, words, ".");
-    #   print "\""words[1]"\","}'
-    _package_modules = [
-        "apk",
-        "apt_key",
-        "apt",
-        "apt_rpm",
-        "bower",
-        "bundler",
-        "composer",
-        "cpanm",
-        "dnf",
-        "easy_install",
-        "flatpak",
-        "flatpak_remote",
-        "gem",
-        "homebrew_cask",
-        "homebrew",
-        "homebrew_tap",
-        "layman",
-        "macports",
-        "maven_artifact",
-        "npm",
-        "openbsd_pkg",
-        "opkg",
-        "package",
-        "pacman",
-        "pear",
-        "pip",
-        "pkg5_publisher",
-        "pkg5",
-        "pkgin",
-        "pkgng",
-        "pkgutil",
-        "portage",
-        "portinstall",
-        "rhn_register",
-        "rpm_key",
-        "slackpkg",
-        "snap",
-        "sorcery",
-        "svr4pkg",
-        "swdepot",
-        "swupd",
-        "urpmi",
-        "xbps",
-        "yarn",
-        "yum",
-        "zypper",
-    ]
-
-    _module_ignore_states = [
-        "absent",
-    ]
-
-    _module_ignore_parameters = {
-        "apt_key": [
-            "data",
-        ],
-        "yum": [
-            "list",
-        ]
-    }
-
-    _package_name_keys = [
-        "name",
-        "package",
-        "pkg",
-        "deb",
-        "key",
-    ]
-
-    def get_package_name(self, action):
-        """Attempt to find package name."""
-        for key in self._package_name_keys:
-            found_package_name = action.get(key)
-            if found_package_name:
-                return found_package_name
-        return found_package_name
-
-    def matchtask(self, file, task):
-        module = task["action"]["__ansible_module__"]
-
-        if module not in self._package_modules:
-            return False
-
-        is_task_retryable = 'until' in task
-        if is_task_retryable:
-            return False
-
-        is_state_whitelisted = task['action'].get('state') in self._module_ignore_states
-        if is_state_whitelisted:
-            return False
-
-        has_whitelisted_parameter = (
-            set(self._module_ignore_parameters.get(module, [])).intersection(set(task['action']))
-        )
-        if has_whitelisted_parameter:
-            return False
-
-        found_package_name = self.get_package_name(task['action'])
-        if not found_package_name:
-            return True
-
-        is_package_file = '.' in found_package_name
-        is_package_html = '://' in found_package_name
-        is_local_package_file = is_package_file and not is_package_html
-        if is_local_package_file:
-            return False
-
-        return True


Test patch:
diff --git a/test/TestPackageHasRetry.py b/test/TestPackageHasRetry.py
deleted file mode 100644
--- a/test/TestPackageHasRetry.py
+++ /dev/null
@@ -1,78 +0,0 @@
-import unittest
-
-from ansiblelint import RulesCollection
-from ansiblelint.rules.PackageHasRetryRule import PackageHasRetryRule
-from test import RunFromText
-
-SUCCESS = '''
----
-- hosts: hosts
-  tasks:
-    - name: "Package has retry test success"
-      apt:
-        pkg: foo
-      register: result
-      until: result is succeeded
-
-    - name: remove software
-      package:
-        name: some_package
-        state: absent
-
-    - name: Install from file - remote
-      apt:
-        deb: https://example.com/python-ppq_0.1-1_all.deb
-        state: present
-      until: result.rc == 0
-
-    - name: Install a .deb package
-      apt:
-        deb: /tmp/mypackage.deb
-
-    - name: install software from a local source
-      package:
-        name: "/tmp/some_package.rpm"
-        state: present
-
-    - name: packaging key
-      apt_key:
-        data: "{{ lookup('file', release_key) }}"
-        id: "{{ release_key_id }}"
-        state: present
-
-    - name: list packages
-      yum:
-        list: some_package
-'''
-
-FAILURE = '''
----
-- hosts: hosts
-  tasks:
-    - name: "Package has retry test failure"
-      apt:
-        package: foo
-    - name: "Package has retry test failure"
-      macports:
-        pkg: foo
-    - name: Install from file - remote
-      apt:
-        deb: https://example.com/python-ppq_0.1-1_all.deb
-        state: present
-'''
-
-
-class TestPackageHasRetry(unittest.TestCase):
-    collection = RulesCollection()
-    collection.register(PackageHasRetryRule())
-
-    def setUp(self):
-        self.runner = RunFromText(self.collection)
-
-    def test_file_positive(self):
-        results = self.runner.run_playbook(SUCCESS)
-        self.assertEqual(0, len(results))
-
-    def test_file_negative(self):
-        results = self.runner.run_playbook(FAILURE)
-        self.assertEqual(3, len(results))