Repository: ansible/ansible

Issue:
Apt module failing to update cache in Ubuntu 16.04 LTS
_From @tjbenator on April 23, 2016 7:18_

<!--- Verify first that your issue/request is not already reported in GitHub -->
##### ISSUE TYPE

<!--- Pick one below and delete the rest: -->
- Bug Report
##### COMPONENT NAME
- apt
##### ANSIBLE VERSION

```
ansible 2.0.1.0
```
##### CONFIGURATION

Mostly defaults, nothing that would affect this.
https://github.com/binarypenguin/automation/blob/master/ansible.cfg
##### OS / ENVIRONMENT

Ubuntu 16.04 -> Ubuntu 16.04
##### SUMMARY

<!--- Explain the problem briefly -->
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

```
ansible [HOSTNAME] -m apt -a 'update_cache=yes cache_valid_time=3600'
```
##### EXPECTED RESULTS

Download apt updates
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with high verbosity (-vvvv) -->

```
HOSTNAME | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "cache_valid_time": 3600, 
            "deb": null, 
            "default_release": null, 
            "dpkg_options": "force-confdef,force-confold", 
            "force": false, 
            "install_recommends": null, 
            "package": null, 
            "purge": false, 
            "state": "present", 
            "update_cache": true, 
            "upgrade": null
        }, 
        "module_name": "apt"
    }, 
    "msg": "Could not fetch updated apt files"
}
```

`sudo apt-get update` works fine on the host. I thought the issue was that Ubuntu 16.04 LTS does not include Aptitude by default. Also python-apt wasn't installed. Installing them both on the remote machine did not fix the issue.


_Copied from original issue: ansible/ansible-modules-core#3523_


Discussion:
_From @koenpunt on May 29, 2016 16:2_

For me installing the `python-minimal` and `aptitude` packages was enough, but it would be nice if Ansible just used the `apt` command.

_From @ansibot on July 31, 2016 22:16_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/jctanner/ansibullbot/blob/ISSUE_TRIAGE_CLASS/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

_From @subcreator on September 2, 2016 10:20_

I have the same problem!

_From @ansibot on September 2, 2016 10:28_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

_From @resuni on September 16, 2016 3:40_

Experiencing the same problem on Debian 8 with Ansible 2.1.1.0. 

> For me installing the python-minimal and aptitude packages was enough, but it would be nice if Ansible just used the apt command.

Should probably use `apt-get` instead. `apt` will throw a warning, even if just passing it through grep or sed:

```
$ apt search test | grep hello

WARNING: apt does not have a stable CLI interface yet. Use with caution in scripts.
```

_From @zezke on September 20, 2016 13:0_

I can verify that install aptitude did work, but it brought along some extra packages as well. Would be nice if all that could be avoided by making Ansible use `apt-get` instead.

_From @ansibot on September 20, 2016 13:17_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

_From @ansibot on October 6, 2016 14:44_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

_From @codegastudio on October 19, 2016 13:34_

Same problem with raspberry pi on jessie
update_cache option looks to be the problem

> ansible 2.1.2.0

_From @Pkuutn on October 20, 2016 13:49_

I have the same issue with Debian Stretch
Ansible 2.1.1

_From @ansibot on October 22, 2016 8:56_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

_From @gerhard-tinned on October 23, 2016 18:49_

+1

I have the same issue. Interestingly, my Ubuntu servers work without problem. my Debian host on the other hand shows exactly this issue. 

_From @ansibot on November 6, 2016 9:51_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

_From @bdgscotland on November 22, 2016 9:28_

+1 on AWS Ubuntu 16.04 LTS AMI and Ansible 2.2.0.0
Workaround for me:
  pre_tasks:
  - name: 'install python2'
    raw: sudo apt-get -y install python-simplejson

  - name: 'install aptitude'
    raw: sudo apt-get -y install aptitude

_From @ansibot on November 22, 2016 9:59_

@ansible, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: issue_renotify_maintainer --->
_From @bekerov on December 4, 2016 18:1_

same trouble. i cant figure out. Installed today ansible on new machine and my ansible is not work
_From @ansibot on December 7, 2016 20:4_

This repository has been locked. All new issues and pull requests should be filed in https://github.com/ansible/ansible

Please read through the <a href="http://docs.ansible.com/ansible/dev_guide/repomerge.html#move-issues-and-prs-to-new-repo">repomerge</a> page in the dev guide. The guide contains links to tools which automatically move your issue or pull request to the ansible/ansible repo.

<!-- boilerplate: repomerge -->
heya ppl!
after doing some research I've conculded that error appears when there are
some problems wit repositories or repository settings.
i.e. i had a stalled google chrome repo config in /etc/apt/sources.list.d
First thing you should do is ssh and check the result of apt-get update
command manually

2016-12-08 17:00 GMT+03:00 YPCrumble <notifications@github.com>:

> *From @gerhard-tinned <https://github.com/gerhard-tinned> on October 23,
> 2016 18:49*
>
> +1
>
> I have the same issue. Interestingly, my Ubuntu servers work without
> problem. my Debian host on the other hand shows exactly this issue.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/ansible/ansible/issues/18987#issuecomment-265745901>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AFWX0lkJTMQPyttQLXBhF5Ht3lEY0v-2ks5rGA1igaJpZM4LH2PK>
> .
>

I am having this exact issue, with Ubuntu 16.04 and `ansible 2.2.0.0`.
I had same issue using APT | Update on Debian 7.
So when figuring out, it showed me that there are /etc/sources.list.d/*.list configs are changed and the apt-update can't locate those deb mirrors.
When I moved or removed from the location and hitting again the play, it worked fine with my case.
Hope it helps. And Thanks man for pointing out at .list configs @Pkuutn. 
The weirdest thing about this actually is that the following fails:

```
- apt: update_cache=yes upgrade=safe cache_valid_time=3600
```

But this does work:

```
- apt: name=aptitude state=present
- apt: update_cache=yes upgrade=safe cache_valid_time=3600
```
@koenpunt That makes perfect sense, no? Since the issue is that a `safe-upgrade` depends on `aptitude` which is installed by your first task. This is the current work-around.
I would think that the apt task always uses the same tool in the background, but apparently it's not.
I am also experiencing the "Could not fetch updated apt files" error when `update_cache=yes` is set.
same here with ubuntu 16.04 lts and ansible 2.2.1.0

Same issue on ansible 2.2.1.0
Same issue here. @koenpunt's workaround seems to have helped!
@YPCrumble did you ever find a workaround? aptitude didn't help for me also. running everything as raw didn't either
Are there plans to address this issue?  The current workaround, for me, is to run the following command before using Ansible:

`sudo apt-get install -y aptitude software-properties-common`

I use Packer with Ansible, so it's fairly straight forward to put that into a script that bootstraps Ansible on Ubuntu but it's a really ugly hack.
I'm having this exact issue, with Ubuntu 16.04 and ansible 2.2.0.0. 
When I run apt commands on my host machine I get no error
@misham luckily you can work this around within Ansible, as only certain `apt` features require `aptitude` (namely upgrade) to work. So instead of

```
- name: "Upgrade existing packages"
  apt: 
    update_cache: yes 
    upgrade: yes 
```

which would fail due to lack of `aptitude`, you change your playbook to satisfy Ansible needs first, then you do your remaining tasks:

```
- name: "Working around Ansible lameness..."
  apt:
    pkg: "{{ item }}"
    state: latest
    update_cache: yes
    cache_valid_time: 3600
  with_items:
    - aptitude

- name: "Upgrade existing packages"
  apt: 
    update_cache: yes 
    upgrade: yes 
```

and you are good as before.
cc @mgwilliams
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Also getting this on version 2.3.1.0....

Tried using @MarcinOrlowski solution but that hasn't worked for me and produces a huge error message

https://pastebin.com/Q9JHXX0y  <-- Yeah It's that big!!
The behavior of `aptitude safe-upgrade` (according to its man page) is
- update existing packages
- install new packages where necessary for upgrading existing packages
- remove packages which are unused

and the second point seems to be the reason for using `aptitude`, cf. #2540

With `apt-get`, it seems like we would have to run
```
apt-get upgrade --with-new-pkgs --autoremove
```
where the options induce the expected & desired behavior. This is again according to the man page, and also [this askubuntu question](https://askubuntu.com/questions/117088/aptitude-safe-upgrade-equivalence-with-apt-get))

So... I'm gonna submit a pull request where the `upgrade: yes/safe` induce the above `apt-get` command. Also, `upgrade: full` will become synonymous with `upgrade: dist` and both will trigger `apt-get dist-upgrade`. 

An argument in favor of `apt-get` over `aptitude` is that the former is more back-end, while the latter is more user-friendly - I suppose (peronsal humble opinion) that in ansible core, we should try to use the potentially more stable and backwards compatible back-end tools.
@GuruTom I only took a *very* quick look at your pasetbin, but the fact that the output contains
```
No apport report written because the error message indicates its a followup error from a previous failure
```
is (at least for me) an indicator that this might be due to a pre-existing condition ; )

Looking a bit further, I see 
```
stdout: No space left on device
```
Wild guess: Is it possible that your boot partition is full of kernel images?
The guess is based on these lines:
```
mkinitramfs failure cpio 141 gzip 1\r\n
update-initramfs: failed for /boot/initrd.img-4.4.0-87-generic with 1.\r\n
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1\r\n
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-87-generic.postinst line 1052.\r\n
dpkg: error processing package linux-image-4.4.0-87-generic (--configure):\r\n
subprocess installed post-installation script returned error exit status 2\r\n
dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-87-generic:\r\n
linux-image-extra-4.4.0-87-generic depends on linux-image-4.4.0-87-generic; however:\r\n 
Package linux-image-4.4.0-87-generic is not configured yet.
```
I think there might be something wrong with your system, @MarcinOrlowski's solution should fix it.

Resolving patch:
diff --git a/lib/ansible/modules/packaging/os/apt.py b/lib/ansible/modules/packaging/os/apt.py
--- a/lib/ansible/modules/packaging/os/apt.py
+++ b/lib/ansible/modules/packaging/os/apt.py
@@ -89,6 +89,7 @@
       - 'If full, performs an aptitude full-upgrade.'
       - 'If dist, performs an apt-get dist-upgrade.'
       - 'Note: This does not upgrade a specific package, use state=latest for that.'
+      - 'Note: Since 2.4, apt-get is used as a fall-back if aptitude is not present.'
     version_added: "1.1"
     required: false
     default: "no"
@@ -126,14 +127,21 @@
     required: false
     default: false
     version_added: "2.1"
+  force_apt_get:
+    description:
+      - Force usage of apt-get instead of aptitude
+    required: false
+    default: false
+    version_added: "2.4"
 requirements:
    - python-apt (python 2)
    - python3-apt (python 3)
-   - aptitude
+   - aptitude (before 2.4)
 author: "Matthew Williams (@mgwilliams)"
 notes:
-   - Three of the upgrade modes (C(full), C(safe) and its alias C(yes)) require C(aptitude), otherwise
-     C(apt-get) suffices.
+   - Three of the upgrade modes (C(full), C(safe) and its alias C(yes))
+     required C(aptitude) up to 2.3, since 2.4 C(apt-get) is used as a
+     fall-back.
 '''
 
 EXAMPLES = '''
@@ -711,6 +719,7 @@ def cleanup(m, purge=False, force=False, operation=None,
 
 
 def upgrade(m, mode="yes", force=False, default_release=None,
+            use_apt_get=False,
             dpkg_options=expand_dpkg_options(DPKG_OPTIONS)):
     if m.check_mode:
         check_arg = '--simulate'
@@ -719,19 +728,23 @@ def upgrade(m, mode="yes", force=False, default_release=None,
 
     apt_cmd = None
     prompt_regex = None
-    if mode == "dist":
+    if mode == "dist" or (mode == "full" and use_apt_get):
         # apt-get dist-upgrade
         apt_cmd = APT_GET_CMD
         upgrade_command = "dist-upgrade"
-    elif mode == "full":
+    elif mode == "full" and not use_apt_get:
         # aptitude full-upgrade
         apt_cmd = APTITUDE_CMD
         upgrade_command = "full-upgrade"
     else:
-        # aptitude safe-upgrade # mode=yes # default
-        apt_cmd = APTITUDE_CMD
-        upgrade_command = "safe-upgrade"
-        prompt_regex = r"(^Do you want to ignore this warning and proceed anyway\?|^\*\*\*.*\[default=.*\])"
+        if use_apt_get:
+            apt_cmd = APT_GET_CMD
+            upgrade_command = "upgrade --with-new-pkgs --autoremove"
+        else:
+            # aptitude safe-upgrade # mode=yes # default
+            apt_cmd = APTITUDE_CMD
+            upgrade_command = "safe-upgrade"
+            prompt_regex = r"(^Do you want to ignore this warning and proceed anyway\?|^\*\*\*.*\[default=.*\])"
 
     if force:
         if apt_cmd == APT_GET_CMD:
@@ -860,6 +873,7 @@ def main():
             autoremove=dict(type='bool', default='no'),
             autoclean=dict(type='bool', default='no'),
             only_upgrade=dict(type='bool', default=False),
+            force_apt_get=dict(type='bool', default=False),
             allow_unauthenticated=dict(default='no', aliases=['allow-unauthenticated'], type='bool'),
         ),
         mutually_exclusive=[['package', 'upgrade', 'deb']],
@@ -894,8 +908,11 @@ def main():
     if p['upgrade'] == 'no':
         p['upgrade'] = None
 
-    if not APTITUDE_CMD and p.get('upgrade', None) in ['full', 'safe', 'yes']:
-        module.fail_json(msg="Could not find aptitude. Please ensure it is installed.")
+    use_apt_get = p['force_apt_get']
+
+    if not use_apt_get and not APTITUDE_CMD and p.get('upgrade', None) in ['full', 'safe', 'yes']:
+        module.warn("Could not find aptitude. Using apt-get instead")
+        use_apt_get = True
 
     updated_cache = False
     updated_cache_time = 0
@@ -956,7 +973,7 @@ def main():
         force_yes = p['force']
 
         if p['upgrade']:
-            upgrade(module, p['upgrade'], force_yes, p['default_release'], dpkg_options)
+            upgrade(module, p['upgrade'], force_yes, p['default_release'], use_apt_get, dpkg_options)
 
         if p['deb']:
             if p['state'] != 'present':
@@ -976,7 +993,7 @@ def main():
         if latest and all_installed:
             if packages:
                 module.fail_json(msg='unable to install additional packages when ugrading all installed packages')
-            upgrade(module, 'yes', force_yes, p['default_release'], dpkg_options)
+            upgrade(module, 'yes', force_yes, p['default_release'], use_apt_get, dpkg_options)
 
         if packages:
             for package in packages:


Test patch:
diff --git a/test/integration/inventory b/test/integration/inventory
--- a/test/integration/inventory
+++ b/test/integration/inventory
@@ -10,6 +10,9 @@ facthost[0:20] ansible_host=1270.0.0.1 ansible_connection=local
 [binary_modules]
 testhost_binary_modules ansible_host=127.0.0.1 ansible_connection=local
 
+[local_group]
+kube-pippin.knf.local
+
 # the following inline declarations are accompanied
 # by (preferred) group_vars/ and host_vars/ variables
 # and are used in testing of variable precedence
diff --git a/test/integration/targets/apt/tasks/main.yml b/test/integration/targets/apt/tasks/main.yml
--- a/test/integration/targets/apt/tasks/main.yml
+++ b/test/integration/targets/apt/tasks/main.yml
@@ -24,20 +24,49 @@
  - include: 'apt-builddep.yml'
    when: ansible_distribution in ('Ubuntu', 'Debian')
 
- - name: Check if aptitude is installed
-   command: dpkg-query -l aptitude
-   register: deb_check
+ - include: upgrade.yml upgrade_type=dist
    when: ansible_distribution in ('Ubuntu', 'Debian')
 
- - include: upgrade.yml upgrade_type=dist
+ - name: Check if aptitude is installed
+   command: dpkg-query --show --showformat='${db:Status-Abbrev}' aptitude
+   register: aptitude_status
    when: ansible_distribution in ('Ubuntu', 'Debian')
 
- - include: upgrade.yml upgrade_type=safe
+ - debug: var=aptitude_status.stdout
+
+ - name: Remove aptitude, if installed, to test fall-back to apt-get
+   apt:
+     pkg: aptitude
+     state: absent
+   when:
+     - aptitude_status.stdout.find('ii') != -1
+
+ - include: "upgrade.yml aptitude_present={{ False | bool }} upgrade_type={{ item.upgrade_type }} force_apt_get={{ item.force_apt_get }}"
    when:
      - ansible_distribution in ('Ubuntu', 'Debian')
-     - deb_check.stdout.find('no packages found') != -1
+   with_items:
+     - { upgrade_type: safe, force_apt_get: False }
+     - { upgrade_type: full, force_apt_get: False }
+     - { upgrade_type: safe, force_apt_get: True }
+     - { upgrade_type: full, force_apt_get: True }
+
+ - name: (Re-)Install aptitude, run same tests again
+   apt:
+     pkg: aptitude
+     state: present
 
- - include: upgrade.yml upgrade_type=full
+ - include: upgrade.yml upgrade_type={{ item.upgrade_type }} force_apt_get={{ item.force_apt_get }}
    when:
      - ansible_distribution in ('Ubuntu', 'Debian')
-     - deb_check.stdout.find('no packages found') != -1
+   with_items:
+     - { upgrade_type: safe, force_apt_get: False }
+     - { upgrade_type: full, force_apt_get: False }
+     - { upgrade_type: safe, force_apt_get: True }
+     - { upgrade_type: full, force_apt_get: True }
+
+ - name: Remove aptitude if not originally present
+   apt:
+     pkg: aptitude
+     state: absent
+   when:
+     - aptitude_status.stdout.find('ii') == -1
diff --git a/test/integration/targets/apt/tasks/upgrade.yml b/test/integration/targets/apt/tasks/upgrade.yml
--- a/test/integration/targets/apt/tasks/upgrade.yml
+++ b/test/integration/targets/apt/tasks/upgrade.yml
@@ -14,14 +14,35 @@
     that:
       - "{{ hello_version.stdout }}=={{ hello_old_version }}"
 
-- name: "(upgrade type: {{upgrade_type}}) upgrade packages to latest version"
+- name: "(upgrade type: {{upgrade_type}}) upgrade packages to latest version, force_apt_get: {{force_apt_get|default(False)}}"
   apt:
     upgrade: "{{ upgrade_type }}"
+    force_apt_get: "{{ force_apt_get | default(False) }}"
+  register: upgrade_result
 
 - name: check hello version
   shell: dpkg -s hello | grep Version | awk '{print $2}'
   register: hello_version
 
+- debug: var=upgrade_result.warnings|default([])
+- debug: var=aptitude_present
+- debug: var=force_apt_get
+
+- name: check that warning is not given when force_apt_get set
+  assert:
+    that:
+      - "'Could not find aptitude. Using apt-get instead' not in upgrade_result.warnings | default([])"
+  when:
+    - force_apt_get | default(False)
+
+- name: check that warning is given when aptitude not found and force_apt_get not set
+  assert:
+    that:
+      - "'Could not find aptitude. Using apt-get instead' in upgrade_result.warnings"
+  when:
+    - not aptitude_present|default(True)
+    - not force_apt_get|default(False)
+
 - name: check that old version upgraded correctly
   assert:
     that:
@@ -31,12 +52,13 @@
 - name: "(upgrade type: {{upgrade_type}}) upgrade packages to latest version (Idempotant)"
   apt:
     upgrade: "{{ upgrade_type }}"
-  register: result
+    force_apt_get: "{{ force_apt_get | default(False) }}"
+  register: second_upgrade_result
 
 - name: check that nothing has changed (Idempotant)
   assert:
     that:
-      - "result.changed == false"
+      - "second_upgrade_result.changed == false"
 
 - name: remove hello
   apt: