Repository: ansible/ansible

Issue:
uri module can not find auth data in ~/.netrc if there is a port number set explicitly in the url
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME

uri

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.1.0
  config file = /home/bookwar/playbooks/ansible.cfg
  configured module search path = [u'/home/bookwar/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.14 (default, Nov  2 2017, 18:42:05) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)]
```

##### CONFIGURATION
none

##### OS / ENVIRONMENT

Fedora 27

##### SUMMARY

uri module can not use authentication via .netrc file, if `url` contains port specification.

When parsing .netrc file for username and password in https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/urls.py#L818, we search for the `parsed[1]` string. But it contains port specification (=`localhost:8081`), while .netrc contains only hostnames (=`localhost`).

##### STEPS TO REPRODUCE
1. configure .netrc on the target host
```
machine localhost
login admin
password admin123
```
2. Check that curl works on the target host
```
curl --netrc -v http://localhost:8081/service/siesta/rest/v1/script/
```
3. Run `uri` task on the target host
```yaml

- name: Run curl request to the host with explicitly specified port
  uri:
    url: "http://localhost:8081/service/siesta/rest/v1/script/"
    method: GET
    status_code: 204
```
##### EXPECTED RESULTS

Ansible task works and produces the same result as curl command.

##### ACTUAL RESULTS

Curl works, while ansible gets `403 Forbidden` error.

##### Workaround

If ~/.netrc is set to:
```
machine localhost:8081
login admin
password admin123
```
ansible task starts to work. But then `curl --netrc` fails.


Discussion:
Files identified in the description:
* [lib/ansible/modules/net_tools/basics/uri.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/net_tools/basics/uri.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @romeotheriault
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -364,13 +364,28 @@ def _build_https_connection(self, host, **kwargs):
         return httplib.HTTPSConnection(host, **kwargs)
 
 
+class ParseResultDottedDict(dict):
+    '''
+    A dict that acts similarly to the ParseResult named tuple from urllib
+    '''
+    def __init__(self, *args, **kwargs):
+        super(ParseResultDottedDict, self).__init__(*args, **kwargs)
+        self.__dict__ = self
+
+    def as_list(self):
+        '''
+        Generate a list from this dict, that looks like the ParseResult named tuple
+        '''
+        return [self.get(k, None) for k in ('scheme', 'netloc', 'path', 'params', 'query', 'fragment')]
+
+
 def generic_urlparse(parts):
     '''
     Returns a dictionary of url parts as parsed by urlparse,
     but accounts for the fact that older versions of that
     library do not support named attributes (ie. .netloc)
     '''
-    generic_parts = dict()
+    generic_parts = ParseResultDottedDict()
     if hasattr(parts, 'netloc'):
         # urlparse is newer, just read the fields straight
         # from the parts object
@@ -729,16 +744,14 @@ def http_request(self, req):
 
 
 def maybe_add_ssl_handler(url, validate_certs):
-    # FIXME: change the following to use the generic_urlparse function
-    #        to remove the indexed references for 'parsed'
-    parsed = urlparse(url)
-    if parsed[0] == 'https' and validate_certs:
+    parsed = generic_urlparse(urlparse(url))
+    if parsed.scheme == 'https' and validate_certs:
         if not HAS_SSL:
             raise NoSSLError('SSL validation is not available in your version of python. You can use validate_certs=False,'
                              ' however this is unsafe and not recommended')
 
         # do the cert validation
-        netloc = parsed[1]
+        netloc = parsed.netloc
         if '@' in netloc:
             netloc = netloc.split('@', 1)[1]
         if ':' in netloc:
@@ -767,10 +780,8 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
     if ssl_handler:
         handlers.append(ssl_handler)
 
-    # FIXME: change the following to use the generic_urlparse function
-    #        to remove the indexed references for 'parsed'
-    parsed = urlparse(url)
-    if parsed[0] != 'ftp':
+    parsed = generic_urlparse(urlparse(url))
+    if parsed.scheme != 'ftp':
         username = url_username
 
         if headers is None:
@@ -778,20 +789,20 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
 
         if username:
             password = url_password
-            netloc = parsed[1]
-        elif '@' in parsed[1]:
-            credentials, netloc = parsed[1].split('@', 1)
+            netloc = parsed.netloc
+        elif '@' in parsed.netloc:
+            credentials, netloc = parsed.netloc.split('@', 1)
             if ':' in credentials:
                 username, password = credentials.split(':', 1)
             else:
                 username = credentials
                 password = ''
 
-            parsed = list(parsed)
-            parsed[1] = netloc
+            parsed_list = parsed.as_list()
+            parsed_list[1] = netloc
 
             # reconstruct url without credentials
-            url = urlunparse(parsed)
+            url = urlunparse(parsed_list)
 
         if username and not force_basic_auth:
             passman = urllib_request.HTTPPasswordMgrWithDefaultRealm()
@@ -815,7 +826,7 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
         else:
             try:
                 rc = netrc.netrc(os.environ.get('NETRC'))
-                login = rc.authenticators(parsed[1])
+                login = rc.authenticators(parsed.hostname)
             except IOError:
                 login = None
 


Test patch:
diff --git a/test/integration/targets/uri/tasks/main.yml b/test/integration/targets/uri/tasks/main.yml
--- a/test/integration/targets/uri/tasks/main.yml
+++ b/test/integration/targets/uri/tasks/main.yml
@@ -373,3 +373,17 @@
   register: result
   failed_when: result is not failed
   when: has_httptester
+
+- name: Write out netrc template
+  template:
+    src: netrc.j2
+    dest: "{{ output_dir }}/netrc"
+
+- debug:
+    msg: "{{ lookup('file', output_dir ~ '/netrc') }}"
+
+- name: Test netrc with port
+  uri:
+    url: "https://{{ httpbin_host }}:443/basic-auth/user/passwd"
+  environment:
+    NETRC: "{{ output_dir|expanduser }}/netrc"
diff --git a/test/integration/targets/uri/templates/netrc.j2 b/test/integration/targets/uri/templates/netrc.j2
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/uri/templates/netrc.j2
@@ -0,0 +1,3 @@
+machine {{ httpbin_host }}
+login user
+password passwd