Repository: ansible/ansible

Issue:
Apt module fails to download deb package with 403 http error
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
apt

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel c8ba8bdd6d) last updated 2017/06/29 13:05:30 (GMT +200)
  python version = 2.7.10 (default, Feb  7 2017, 00:08:15) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
host_key_checking = False
pipelining = True

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
macOS Sierra 10.12.5 (Client) / Ubuntu 16.04 (Server)

##### SUMMARY
<!--- Explain the problem briefly -->
Apt module can't download deb package.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Install wkhtmltopdf package
  hosts: "{{ target }}"
  become: yes

  tasks:
    - name: Install
      apt:
        deb: https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Deb is downloaded and installed.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Download fails with 403 http error.

<!--- Paste verbatim command output between quotes below -->
```
TASK [wkhtmltopdf : Install] ****************************************************************************************************************
task path: /***/provisioner/roles/wkhtmltopdf/tasks/main.yml:7
Using module file /***/ansible/ansible/lib/ansible/modules/packaging/os/apt.py
<x.x.x.x> ESTABLISH SSH CONNECTION FOR USER: None
<x.x.x.x> SSH: EXEC ssh -vvv -F ./ssh.conf -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=20 -o 'ControlPath=~/.ssh/connection-***-ansible-%r@%h:%p' x.x.x.x '/bin/sh -c '"'"'sudo sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ujruezrxyszodncqvbguxciqefxuzrut; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<x.x.x.x> (1, '\n{"msg": "Failed to download https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb, HTTP Error 403: Forbidden", "failed": true, "changed": false, "invocation": {"module_args": {"dpkg_options": "force-confdef,force-confold", "autoremove": false, "force": false, "package": null, "autoclean": false, "purge": false, "allow_unauthenticated": false, "state": "present", "upgrade": null, "update_cache": null, "deb": "https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb", "only_upgrade": false, "cache_valid_time": 0, "default_release": null, "install_recommends": null}}}\n', 'OpenSSH_7.4p1, LibreSSL 2.5.0\r\ndebug1: Reading configuration data ./ssh.conf\r\ndebug1: ./ssh.conf line 1: Applying options for x.x.x.*\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 35871\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
fatal: [x.x.x.x]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "allow_unauthenticated": false,
            "autoclean": false,
            "autoremove": false,
            "cache_valid_time": 0,
            "deb": "https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb",
            "default_release": null,
            "dpkg_options": "force-confdef,force-confold",
            "force": false,
            "install_recommends": null,
            "only_upgrade": false,
            "package": null,
            "purge": false,
            "state": "present",
            "update_cache": null,
            "upgrade": null
        }
    }
}

MSG:

Failed to download https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb, HTTP Error 403: Forbidden
```



Discussion:
some more debug infos
~~~
 $ curl -I https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
HTTP/2 301 
date: Fri, 30 Jun 2017 17:33:59 GMT
content-type: text/html
set-cookie: __cfduid=db326f78ed9269f9b0d52c0a7427580261498844039; expires=Sat, 30-Jun-18 17:33:59 GMT; path=/; domain=.wkhtmltopdf.org; HttpOnly
location: https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
strict-transport-security: max-age=63072000
x-frame-options: DENY
x-content-type-options: nosniff
server: cloudflare-nginx
cf-ray: 3772dfae9c323e7a-ZRH
~~~
Hey guys! Any updates on the issue? Seems like the module can't handle redirects. In my case the request is redirected to github which than redirects to one of it's storage servers.
cc @mgwilliams
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
This is all about https://downloads.wkhtmltopdf.org/ being picky about User-Agents.

```bash
andreas@corrino:~$ curl -I https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
HTTP/2 301 
date: Wed, 19 Jul 2017 17:42:32 GMT
content-type: text/html
set-cookie: __cfduid=d7466cfdada1954f8c3edd4bd5017ab771500486152; expires=Thu, 19-Jul-18 17:42:32 GMT; path=/; domain=.wkhtmltopdf.org; HttpOnly
location: https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
strict-transport-security: max-age=63072000
x-frame-options: DENY
x-content-type-options: nosniff
server: cloudflare-nginx
cf-ray: 380f7a528e4a7277-AMS

andreas@corrino:~$
```

vs.

```
andreas@corrino:~$ curl --user-agent "Python-urllib/2.7" -I https://downloads.wkhtmltopdf.org/0.12/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
HTTP/2 403 
date: Wed, 19 Jul 2017 17:43:11 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d2181cf7df8281d47781c2abda13343361500486191; expires=Thu, 19-Jul-18 17:43:11 GMT; path=/; domain=.wkhtmltopdf.org; HttpOnly
cache-control: max-age=10
expires: Wed, 19 Jul 2017 17:43:21 GMT
x-frame-options: SAMEORIGIN
server: cloudflare-nginx
cf-ray: 380f7b460b190737-AMS

andreas@corrino:~$
```

The redirects are not an issue. Both the _apt_ module and the _get_url_ module uses the same `fetch_url` helper function, and _get_url_ has no problem downloading the .deb file. The difference being that the _get_url_ module sets an explicit User-Agent, which is apparently more acceptable. That User-Agent being _ansible-httpget_, coming from the `url_argument_spec` helper function.


Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -1020,7 +1020,7 @@ def fetch_url(module, url, data=None, headers=None, method=None,
 
     username = module.params.get('url_username', '')
     password = module.params.get('url_password', '')
-    http_agent = module.params.get('http_agent', None)
+    http_agent = module.params.get('http_agent', 'ansible-httpget')
     force_basic_auth = module.params.get('force_basic_auth', '')
 
     follow_redirects = module.params.get('follow_redirects', 'urllib2')


Test patch: