Repository: ansible/ansible-modules-core

Issue:
get_url hides netrc failures 
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

`get_url` from `ansible-modules-core` @ 22ce32dd93874b7722762a2c38004c15efbf893c
##### ANSIBLE VERSION

ansible 2.2.0 (258d6fcddf51d1abcc7bfea4e1304aaf4d67d67c)
##### CONFIGURATION

N/A
##### OS / ENVIRONMENT

N/A
##### SUMMARY

Calls to `get_url` can fail because of problems with `$HOME/.netrc`,
and when fetching `file://` URLs these errors are not correctly
reported to the user.
##### STEPS TO REPRODUCE

Create a `.netrc` file with open permissions:

```
echo machine host.example.com login you@example.com password secret \
  > $HOME/.netrc
chmod 664 $HOME/.netrc
```

Try fetching a `file://` URL:

```
- hosts: localhost
  gather_facts: false
  tasks:

    - get_url:
        url: file:///etc/motd
        dest: "./destfile"
        force: true
```

Compare this to fetching an `http://` URL:

```
- hosts: localhost
  gather_facts: false
  tasks:

    - get_url:
        url: "http://google.com/"
        dest: "./destfile"
        force: true
```
##### EXPECTED RESULTS

I expect ansible to fail with a warning about the permissions on the
`$HOME/.netrc` file:

```
fatal: [localhost]: FAILED! => {"changed": false, "dest":
"./destfile", "failed": true, "gid": 1000, "group": "lars",
"mode": "0664", "msg": "Request failed", "owner": "lars",
"response": "An unknown error occurred: ~/.netrc access too
permissive: access permissions must restrict access to only the
owner (/home/lars/.netrc, line 1)", "secontext":
"unconfined_u:object_r:user_home_t:s0", "size": 12, "state":
"file", "status_code": -1, "uid": 1000, "url":
"http://google.com/"}
```
##### ACTUAL RESULTS

Ansible fails with an error that doesn't seem to mean anything to
anybody:

```
fatal: [localhost]: FAILED! => {"changed": false, "failed": true,
"msg": "failed to create temporary content file: 'NoneType' object
has no attribute 'read'"}
```



Discussion:
Looks like netrc is only consulted if the NETRC env variable exists and if the request request http basic auth and no username is provided. https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/urls.py#L768

I'm not sure if the above behavior is expected, but it seems reasonable. How do other apps handle netrc for cases like this?

Any different behavior with force_basic_auth set? Or url_username?

@alikins You can see the reproducer there in my report; it doesn't require setting the `NETRC` environment variable nor does it require basic auth.  It presents with both unauthenticated `file://` and `http://` URLs, and in the case of `file://` URLs the error is reported incorrectly.  This latter condition is neither reasonable nor expected.

The failure mode for `http://` URLs seems reasonable, because it clearly identifies the problem.

I'm getting the same error message, I can't tell what is wrong:

```
fatal: [localhost]: FAILED! => {"changed": false, "dest": "/root/file", "failed": true, "msg": "Request failed", 
"response": "An unknown error occurred: coercing to Unicode: need string or buffer, NoneType found", "state": "absent", 
"status_code": -1, "url": "https://www.rabbitmq.com/releases/erlang/erlang-18.3-1.el7.centos.x86_64.rpm"}
```

with this task:

```
get_url:
  url: "https://www.rabbitmq.com/releases/erlang/erlang-18.3-1.el7.centos.x86_64.rpm"
  dest: "/root/erlang.rpm"
```

Found the error by guessing, apparently does not support https, but there's no way to tell why, because of the error message.

@jpmens, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/jctanner/ansibullbot/blob/ISSUE_TRIAGE_CLASS/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

@jpmens, ping. This issue is still waiting on your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_renotify_maintainer --->

needs_contributor


Resolving patch:
diff --git a/network/basics/get_url.py b/network/basics/get_url.py
--- a/network/basics/get_url.py
+++ b/network/basics/get_url.py
@@ -227,10 +227,14 @@ def url_get(module, url, dest, use_proxy, last_mod_time, force, timeout=10, head
     if info['status'] == 304:
         module.exit_json(url=url, dest=dest, changed=False, msg=info.get('msg', ''))
 
-    # create a temporary file and copy content to do checksum-based replacement
+    # Exceptions in fetch_url may result in a status -1, the ensures a proper error to the user in all cases
+    if info['status'] == -1:
+        module.fail_json(msg=info['msg'], url=url, dest=dest)
+
     if info['status'] != 200 and not url.startswith('file:/') and not (url.startswith('ftp:/') and info.get('msg', '').startswith('OK')):
         module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'], url=url, dest=dest)
 
+    # create a temporary file and copy content to do checksum-based replacement
     if tmp_dest != '':
         # tmp_dest should be an existing dir
         tmp_dest_is_dir = os.path.isdir(tmp_dest)


Test patch: