Repository: ansible/ansible

Issue:
get_url fails when a password is provided in URL
I have tried on ansible 1.6 (devel 5a075e3b93) and 1.5.4 to run the following playbook task (some details changed/removed)

``` yaml
- hosts: tomcat
  vars:
    apps:
      - { app: 'app1', version: '01' }
      - { app: 'app2', version: '05' }
    apps_url: "https://user:password@server.example.com/"

    - name: Download wars
      get_url: url={{ apps_url }}{{ item.app }}.{{ item.version }}.war
        dest=/opt/tomcat/wars/{{ item.app }}.war force=yes validate_certs=no
        mode=0644 owner=root group=root
      with_items: apps
```

The following tasks returns

failed: [tomcat.example.com] => (item={'version': '01', 'app': 'app1'}) => {"dest": "/opt/tomcat/wars/app1.war", "failed": true, "item": {"app": "app1", "version": "01"}, "response": "HTTP Error 401: basic auth failed", "state": "absent", "status_code": 401, "url": "https://user:password@server.example.com/app.01.war"}
msg: Request failed

I tested the url path and credentials by running
`wget --no-check-certificate "https://user:password@server.example.com/app1.01.war` which works without any errors.

I also found that if there are any special characters is the password they need to be [URL encoded](http://en.wikipedia.org/wiki/Percent-encoding) otherwise you will get the error 

`InvalidURL("nonnumeric port: '%s'" % host[i+1:])
httplib.InvalidURL: nonnumeric port`

Is there a way for us to convert the URL in the playbook (maybe jinja filter) by default? I think the [urlencode() jinja filter](http://jinja.pocoo.org/docs/templates/#urlencode) will work but I haven't tested it yet.



Discussion:
Thanks for filing a ticket! I am the friendly GitHub Ansibot. I see you did not fill out your issue description based on our new issue template. Please copy the contents of this [template](https://raw2.github.com/ansible/ansible/devel/examples/issues/ISSUE_TEMPLATE.md) and **paste it into the description** of your ticket. 

Our system will automatically close tickets that do not have an issue template filled out within 7 days. Please note that due to large interest in Ansible, humans may not comment on your ticket if you ask them questions. Don't worry, you're still in the queue and the robots are looking after you.

I cannot reproduce this on either 1.5.4 or the current head of devel. I setup a test server with a password protected directory. The auth fails as expected when I don't have the password correct, I get the same error you're seeing: `"response": "HTTP Error 401: basic auth failed"`, but otherwise it works correctly. So it really seems like the password is incorrect in your url, though I'm not sure why it's working from wget. Does it contain any characters that are non-ASCII?

The password does have non-ASCII characters which is why I tried URL encoding it. Even when I used wget I had to use URL encoding. But I couldn't find a combination that would work with the ansible task. I tried URL encoding, escaping \, and quotes but was unsuccessful.

Any other thoughts on what I can try? Otherwise I will just use command: wget for the task.

@rothgar so the way fetch_url() works internally is to strip the username/password out of the URL and create a basic auth object for urllib. When you're encoding them, they're probably being passed in to that object encoded, which is why you're seeing the 401. So I guess we could decode them in that function, though it might cause problems for users that have % characters in their passwords.

@jimi-c Would it be possible to add a unc_encode=True option to the function to have it optionally encode the password? This is probably a small need and I can look into it myself in the future if need be.

I also found the [urllib.quote function](https://docs.python.org/2/library/urllib.html#urllib.quote) which should be able to do the parsing for us.

Ok, the path I'm going with this is to add two new params: url_username and url_password. This way, there's no need to deal with any encoding/decoding and is easier to use vault-encrypted passwords in the play. I'm nearly ready to post this up to my github clone, and I'll send a PR from there so you can see it/test it.


Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -200,6 +200,8 @@ def url_argument_spec():
         http_agent = dict(default='ansible-httpget'),
         use_proxy = dict(default='yes', type='bool'),
         validate_certs = dict(default='yes', type='bool'),
+        url_username = dict(required=False),
+        url_password = dict(required=False),
     )
 
 
@@ -247,15 +249,22 @@ def fetch_url(module, url, data=None, headers=None, method=None,
             ssl_handler = SSLValidationHandler(module, hostname, port)
             handlers.append(ssl_handler)
 
-    if parsed[0] != 'ftp' and '@' in parsed[1]:
-        credentials, netloc = parsed[1].split('@', 1)
-        if ':' in credentials:
-            username, password = credentials.split(':', 1)
-        else:
-            username = credentials
-            password = ''
-        parsed = list(parsed)
-        parsed[1] = netloc
+    if parsed[0] != 'ftp':
+        url_username = module.params.get('url_username', '')
+        if url_username:
+            username = url_username
+            password = module.params.get('url_password', '')
+            netloc = parsed[1]
+        elif '@' in parsed[1]:
+            credentials, netloc = parsed[1].split('@', 1)
+            if ':' in credentials:
+                username, password = credentials.split(':', 1)
+            else:
+                username = credentials
+                password = ''
+
+            parsed = list(parsed)
+            parsed[1] = netloc
 
         passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
         # this creates a password manager


Test patch: