Repository: ansible/ansible-modules-core

Issue:
apt update_cache gives error on ubuntu for Ansible 2.0.0.2 
##### ISSUE TYPE

Bug Report
##### COMPONENT NAME

apt module
##### ANSIBLE VERSION

N/A
##### SUMMARY

The simple following command:

```
- name: Update apt cache.
  apt: update_cache=true
  become: true
```

Throws an error on AWS Ubuntu instances.

fatal: [xx.xx.xxx.x]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/ubuntu/.ansible/tmp/ansible-tmp-1454536847.17-86172912102079/apt\", line 2630, in <module>\r\n    main()\r\n  File \"/home/ubuntu/.ansible/tmp/ansible-tmp-1454536847.17-86172912102079/apt\", line 603, in main\r\n    cache = apt.Cache()\r\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 107, in **init**\r\n    self.open(progress)\r\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 151, in open\r\n    self._cache = apt_pkg.Cache(progress)\r\nSystemError: E:Encountered a section with no Package: header, E:Problem with MergeList /var/lib/apt/lists/us-west-2.ec2.archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en, E:The package lists or status file could not be parsed or opened.\r\n", "msg": "MODULE FAILURE", "parsed": false}

However, when I ssh into the instance, and do sudo apt-get update manually, there is no problem at all. And then, if I ran the task again, the problem will go away, I guess because I did the manual update on the instance.

Any ideas?

Thanks



Discussion:
I have ran into this problem before too. Someone should look into it. @mgwilliams

Any updates on this?

+1

+1

If i'm not mistaken, by the traceback it looks like this error is occurring within the Python apt module that Ansible uses?

+1
Running `sudo rm -vf /var/lib/apt/lists/*` gets around the problem.

+1
I've stopped using the apt module entirely because of this. Running:

``` yaml
shell: apt-get update
```

is **much** more reliable, so it seems the issue stems from the python apt module like others have pointed out. That being said you will get annoying warnings like this: 

``` sh
[WARNING]: Consider using apt-get module rather than running apt-get
```

```
sudo apt-get clean 
sudo apt-get update
```

this fixed it for me, running it on target machine

@jimi-c  any updates on this?

+1
Ansible 1.9.4
Exactly the same error, doing 'apt-get update' command succeeded while always failed with 3 retries of apt module

Removing /var/lib/apt/lists/\* could resolve this but as command 'sudo apt-get update' works without that cleanup so seems this is due to Ansible apt module.

Right now I have to add a retry with command in case apt module failed

```
- name: apt update
  apt: update_cache=yes cache_valid_time=3600
  register: apt_result
  until: apt_result|success
  retries: 3
  delay: 1
  sudo: yes
  ignore_errors: yes

- name: retry if needed using command apt-get update
  command: apt-get update
  sudo: yes
  when: apt_result|failed
```

@jimi-c 
Problem remains for **Ansible Version 2.0.1.0**, any future plans on this bug?

+1

I've also noticed that if you pass in a `--limit=aSingleHost` it generally works.

using @dohoangkhiem's retry and then `command` trick, I went from ~90% failure to < 1% failure over 130 machines.

+1 yes have this problem also. needs to be fixed

```
- name: hack to resolve Problem with MergeList Issue
  sudo: true
  shell: 'find /var/lib/apt/lists -maxdepth 1 -type f -exec rm -v {} \;'

- apt: update_cache=yes
```

So it still happens in Ansible 2.0.2.0.  Does anisble not even care about this bug? @jimi-c 

+1 
$ ansible --version
ansible 2.0.0.2

Hi,

I am integrating Slack module with Ansible getting same type fo error 

```
TASK [redis : slack] ***********************************************************
fatal: [XX.XX.XX.XX -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}
 [ERROR]: Could not create retry file './ssh/retry/redis-all.retry'. The error was: [Errno 2] No such file or directory: './ssh/retry/redis-all.retry'
```

+1

+1

+1 btw: I'm having this issue running from osx

The same with ansible 2.1; I use @dohoangkhiem thing, but i have few places in my playbook where I deal with the apt (update cache, install java, install docker). So I am thinking to use `shell` instead.

Also using dohoangkhiem's workaround, but it'd be great if this worked reliably.

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/jctanner/ansibullbot/blob/ISSUE_TRIAGE_CLASS/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

Updates??

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

~~+1 on 1.9.6~~

Correction: I had the same error, but it was related to trying to run the same apt task on the same host multiple times (at the same time) if I recall correctly.

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

- Ansible 2.1 from pip

I guess since Ansible has been acquired by RH, we can just forget about apt module support, or have to fix it ourselves.

@akamensky well if they are going to drop support for ubuntu looks like we will have to drop support for ansible.

Since it's so important to you why not fix it and submit a PR?

Clearly this is some edge case with the images provided by ubuntu on amazon (no surprise there, their docker containers suck, too) 

Why not just mitigate this with a task to clear out `/var/lib/apt/lists/*` if you can detect amazon AND ubuntu from facts?

@xenithorb Edge case for any Ubuntu/Debian installation in my case. I've tried Debian 7 netinstall, Ubuntu 14.04 ISO install, I actually haven't tried Amazon ones. But from what you say it look like the issue is also there.

As for making PR -- I did, those are never even looked at.

[Your PR](https://github.com/ansible/ansible/pull/17115/files) has only been published for a month and it has _nothing_ to do with this issue. Which I might add isn't even a bug fix, but an enhancement. (It even links back to a feature_idea tagged issue ansible/ansible#11175). Are you really that surprised they're probably prioritizing bugs over feature ideas?

For a serious bug like this I'm sure if you fixed it you could poke the right people (politely) and perhaps even get on IRC and get something done.

I honestly haven't seen this bug myself, though I admittedly don't touch the debian family side of the house often. When I do though I've noticed stability issues with apt (even sometimes no responsibility of the module itself, the back-end fails contacting a mirror and the logic to recover from that is just bad on their (apt's) behalf. I usually mitigate that with an `until:` + `retries:` loop and let it do its thing. 

+1 On Ubuntu 16.04.1 LTS with ansible 2.0.0.2 and python 2.7

Has nothing to do with Amazon.

sudo rm -rf /var/lib/apt/lists/\* solve the issues momentarily for me too.

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

Do we know who is the maintainer for this? The issue was opened 8 months ago and constantly reported since and it is a core module as we all know. I'd like to know if the maintainer is still available for fixes or we need to find another way.

+1
- Ubuntu 16.10 remote host
- macOS ansible master
  
  ```
  $ ansible-playbook --version
  ansible-playbook 2.1.1.0
  ```
  
  ```
  $ python --version
  Python 2.7.10
  ```

I attempted the workaround suggested [above](https://github.com/ansible/ansible-modules-core/issues/2951#issuecomment-218327951), but the directory doesn't exist

```
FAILED! => {"changed": true, "cmd": "find /var/lib/apt/lists -maxdepth 1 -type f -exec rm -v {} \\;", "delta": "0:00:00.011281", "end": "2016-10-20 22:34:26.408833", "failed": true, "rc": 1, "start": "2016-10-20 22:34:26.397552", "stderr": "find: /var/lib/apt/lists: No such file or directory", "stdout": "", "stdout_lines": [], "warnings": []}
```

The @dohoangkhiem fix was not successful:

```
TASK [apt] *********************************************************************
FAILED - RETRYING: TASK: apt (3 retries left).
FAILED - RETRYING: TASK: apt (2 retries left).
FAILED - RETRYING: TASK: apt (1 retries left).
fatal: [yogax]: FAILED! => {"changed": false, "cmd": "apt-get update", "failed": true, "msg": "[Errno 2] No such file or directory", "rc": 2}
...ignoring

TASK [retry if needed using command apt-get update] ****************************
fatal: [yogax]: FAILED! => {"changed": false, "cmd": "apt-get update", "failed": true, "msg": "[Errno 2] No such file or directory", "rc": 2}
```

This is one of the most annoying issues you can face with ansible, especially since you bump to it very early: practically your very first script will install something using apt. I wasted too much time just to make crazy workarounds that worked only for specific target OS. 

I agree @igorspasic. I was assuming Ansible is more mature...

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

Related:
http://askubuntu.com/questions/561276/my-repos-became-corrupted-unable-to-parse-package-file
https://answers.launchpad.net/ubuntu/+source/apt/+question/267919
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/346386

I am able to reproduce the bug by injecting noise into one of the package list files ...

```
- hosts: u1610
  user: vagrant
  gather_facts: False
  vars:
    pfile: /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages
  tasks:
    - name: screw up a package file
      shell: head -n 100000 /dev/urandom >> {{ pfile }}
      become: true

    - name: Update apt cache.
      apt: update_cache=true
      become: true
```

```
fatal: [u1610]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_name": "apt"
    }, 
    "module_stderr": "OpenSSH_7.2p2, OpenSSL 1.0.2h-fips  3 May 2016\r\ndebug1: Reading configuration data /home/jtanner/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 15640\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to u1610 closed.\r\n", 
    "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_qg6gD1/ansible_module_apt.py\", line 935, in <module>\r\n    main()\r\n  File \"/tmp/ansible_qg6gD1/ansible_module_apt.py\", line 825, in main\r\n    cache = apt.Cache()\r\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 107, in __init__\r\n    self.open(progress)\r\n  File \"/usr/lib/python2.7/dist-packages/apt/cache.py\", line 151, in open\r\n    self._cache = apt_pkg.Cache(progress)\r\nSystemError: E:Unable to parse package file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages (1), E:The package lists or status file could not be parsed or opened.\r\n", 
    "msg": "MODULE FAILURE"
}

```


Resolving patch:
diff --git a/packaging/os/apt.py b/packaging/os/apt.py
--- a/packaging/os/apt.py
+++ b/packaging/os/apt.py
@@ -755,6 +755,31 @@ def get_updated_cache_time():
     return mtimestamp, updated_cache_time
 
 
+# https://github.com/ansible/ansible-modules-core/issues/2951
+def get_cache(module):
+    '''Attempt to get the cache object and update till it works'''
+    cache = None
+    try:
+        cache = apt.Cache()
+    except SystemError:
+        e = get_exception()
+        if '/var/lib/apt/lists/' in str(e).lower():
+            # update cache until files are fixed or retries exceeded
+            retries = 0
+            while retries < 2:
+                (rc, so, se) = module.run_command(['apt-get', 'update', '-q'])
+                retries += 1
+                if rc == 0:
+                    break
+            if rc != 0:
+                module.fail_json(msg='Updating the cache to correct corrupt package lists failed:\n%s\n%s' % (str(e), str(so) + str(se)))    
+            # try again
+            cache = apt.Cache()
+        else:
+            module.fail_json(msg=str(e))
+    return cache
+ 
+
 def main():
     module = AnsibleModule(
         argument_spec = dict(
@@ -821,8 +846,10 @@ def main():
     if p['state'] == 'removed':
         p['state'] = 'absent'
 
+    # Get the cache object
+    cache = get_cache(module)
+
     try:
-        cache = apt.Cache()
         if p['default_release']:
             try:
                 apt_pkg.config['APT::Default-Release'] = p['default_release']


Test patch: