Repository: conda/conda

Issue:
conda does not prompt for proxy username and password 
```
[      COMPLETE      ] |#################################################| 100%
The batch file cannot be found.

C:\Code>conda update conda
Fetching package metadata: .Error: HTTPError: 407 Client Error: Proxy Authentication Required: http://repo.continuum.io/pkgs/pro/win-64/
```



Discussion:
Do you have proxy settings set in your `.condarc` or using the `HTTP_PROXY` environment variable? 

no. I used to have HTTP_PROXY set but it was automatically removed by a company pushed OS update. To avoid reliance on it I enter the proxy id and pw at the prompt.

OK. I guess the new requests code regressed in that it no longer asks for it at a prompt. 

The workaround for now is to just add it to your .condarc (http://conda.pydata.org/docs/config.html), or to set that environment variable. 

I'm trying to understand how the code worked before. Did you always have the proxy server (and port) itself set, and only the username and password was prompted? 

I honestly don't recall entering the proxy info anywhere. The only thing I was ever required to enter was user and pw.

> On May 20, 2014, at 5:39 PM, Aaron Meurer notifications@github.com wrote:
> 
> I'm trying to understand how the code worked before. Did you always have the proxy server (and port) itself set, and only the username and password was prompted?
> 
> \
> Reply to this email directly or view it on GitHub.

I'm still getting this error after entering the proxy info in condarc
The issue really seems to be with the repo. Why am I seeing the "pro" repo? I am using the free distribution of Anaconda. Could this be the problem?

```
C:\>conda update conda
Fetching package metadata: .Error: HTTPError: 407 Client Error: Proxy Authentication Required: http://repo.continuum.io/pkgs/pro/win-64/
```

No, the pro repos are there for everyone, and do not require any authentication on the open internet. 

Ok. I am completely lost since I don't know much about conda. If there is anything that can be tested to provide more information please let me know.

Well I am a little surprised that adding the proxy to .condarc didn't work. Are you sure you used the exact format `http://[username]:[password]@[server]:[port]` (note that the `http://` part is required)?

Also, does it work if you use the `HTTP_PROXY` environment variable, like

```
HTTP_PROXY=yourproxy conda update conda
```

Oh you're on Windows. I don't know if that syntax works on Windows. You may need to 

```
set HTTP_PROXY=yourproxy
conda update conda
```

setting HTTP_PROXY works. 

OK, but setting the http proxy in .condarc to the exact same thing does not work? That's definitely a bug then. 

exact same thing.


Resolving patch:
diff --git a/conda/config.py b/conda/config.py
--- a/conda/config.py
+++ b/conda/config.py
@@ -244,7 +244,10 @@ def get_allowed_channels():
 
 def get_proxy_servers():
     res = rc.get('proxy_servers')
-    if res is None or isinstance(res, dict):
+    if res is None:
+        import requests
+        return requests.utils.getproxies()
+    if isinstance(res, dict):
         return res
     sys.exit("Error: proxy_servers setting not a mapping")
 
diff --git a/conda/connection.py b/conda/connection.py
--- a/conda/connection.py
+++ b/conda/connection.py
@@ -372,3 +372,51 @@ def parse_multipart_files(request):
     buf.seek(0)
 
     return buf
+
+# Taken from urllib3 (actually
+# https://github.com/shazow/urllib3/pull/394). Once it is fully upstreamed to
+# requests.packages.urllib3 we can just use that.
+
+
+def unparse_url(U):
+    """
+    Convert a :class:`.Url` into a url
+
+    The input can be any iterable that gives ['scheme', 'auth', 'host',
+    'port', 'path', 'query', 'fragment']. Unused items should be None.
+
+    This function should more or less round-trip with :func:`.parse_url`. The
+    returned url may not be exactly the same as the url inputted to
+    :func:`.parse_url`, but it should be equivalent by the RFC (e.g., urls
+    with a blank port).
+
+
+    Example: ::
+
+        >>> Url = parse_url('http://google.com/mail/')
+        >>> unparse_url(Url)
+        'http://google.com/mail/'
+        >>> unparse_url(['http', 'username:password', 'host.com', 80,
+        ... '/path', 'query', 'fragment'])
+        'http://username:password@host.com:80/path?query#fragment'
+    """
+    scheme, auth, host, port, path, query, fragment = U
+    url = ''
+
+    # We use "is not None" we want things to happen with empty strings (or 0 port)
+    if scheme is not None:
+        url = scheme + '://'
+    if auth is not None:
+        url += auth + '@'
+    if host is not None:
+        url += host
+    if port is not None:
+        url += ':' + str(port)
+    if path is not None:
+        url += path
+    if query is not None:
+        url += '?' + query
+    if fragment is not None:
+        url += '#' + fragment
+
+    return url
diff --git a/conda/fetch.py b/conda/fetch.py
--- a/conda/fetch.py
+++ b/conda/fetch.py
@@ -15,12 +15,13 @@
 from logging import getLogger
 from os.path import basename, isdir, join
 import sys
-from multiprocessing.pool import ThreadPool
+import getpass
+# from multiprocessing.pool import ThreadPool
 
 from conda import config
 from conda.utils import memoized
-from conda.connection import CondaSession
-from conda.compat import itervalues, get_http_value
+from conda.connection import CondaSession, unparse_url
+from conda.compat import itervalues, get_http_value, input
 from conda.lock import Locked
 
 import requests
@@ -73,20 +74,34 @@ def fetch_repodata(url, cache_dir=None, use_cache=False, session=None):
         headers["If-Modified-Since"] = cache["_mod"]
 
     try:
-        resp = session.get(url + 'repodata.json.bz2', headers=headers)
+        resp = session.get(url + 'repodata.json.bz2', headers=headers, proxies=session.proxies)
         resp.raise_for_status()
         if resp.status_code != 304:
             cache = json.loads(bz2.decompress(resp.content).decode('utf-8'))
 
-    except ValueError:
-        raise RuntimeError("Invalid index file: %srepodata.json.bz2" % url)
+    except ValueError as e:
+        raise RuntimeError("Invalid index file: %srepodata.json.bz2: %s" %
+            (url, e))
 
     except requests.exceptions.HTTPError as e:
+        if e.response.status_code == 407: # Proxy Authentication Required
+            handle_proxy_407(url, session)
+            # Try again
+            return fetch_repodata(url, cache_dir=cache_dir, use_cache=use_cache, session=session)
         msg = "HTTPError: %s: %s\n" % (e, url)
         log.debug(msg)
         raise RuntimeError(msg)
 
     except requests.exceptions.ConnectionError as e:
+        # requests isn't so nice here. For whatever reason, https gives this
+        # error and http gives the above error. Also, there is no status_code
+        # attribute here. We have to just check if it looks like 407.  See
+        # https://github.com/kennethreitz/requests/issues/2061.
+        if "407" in str(e): # Proxy Authentication Required
+            handle_proxy_407(url, session)
+            # Try again
+            return fetch_repodata(url, cache_dir=cache_dir, use_cache=use_cache, session=session)
+
         msg = "Connection error: %s: %s\n" % (e, url)
         stderrlog.info('Could not connect to %s\n' % url)
         log.debug(msg)
@@ -102,10 +117,31 @@ def fetch_repodata(url, cache_dir=None, use_cache=False, session=None):
 
     return cache or None
 
+def handle_proxy_407(url, session):
+    """
+    Prompts the user for the proxy username and password and modifies the
+    proxy in the session object to include it.
+    """
+    # We could also use HTTPProxyAuth, but this does not work with https
+    # proxies (see https://github.com/kennethreitz/requests/issues/2061).
+    scheme = requests.packages.urllib3.util.url.parse_url(url).scheme
+    username, passwd = get_proxy_username_and_pass(scheme)
+    session.proxies[scheme] = add_username_and_pass_to_url(session.proxies[scheme], username, passwd)
+
+def add_username_and_pass_to_url(url, username, passwd):
+    urlparts = list(requests.packages.urllib3.util.url.parse_url(url))
+    urlparts[1] = username + ':' + passwd
+    return unparse_url(urlparts)
+
+def get_proxy_username_and_pass(scheme):
+    username = input("\n%s proxy username: " % scheme)
+    passwd = getpass.getpass("Password:")
+    return username, passwd
+
 @memoized
 def fetch_index(channel_urls, use_cache=False, unknown=False):
     log.debug('channel_urls=' + repr(channel_urls))
-    pool = ThreadPool(5)
+    # pool = ThreadPool(5)
     index = {}
     stdoutlog.info("Fetching package metadata: ")
     session = CondaSession()
@@ -171,15 +207,29 @@ def download(url, dst_path, session=None, md5=None, urlstxt=False):
 
     with Locked(dst_dir):
         try:
-            resp = session.get(url, stream=True)
+            resp = session.get(url, stream=True, proxies=session.proxies)
             resp.raise_for_status()
-        except IOError:
-            raise RuntimeError("Could not open '%s'" % url)
         except requests.exceptions.HTTPError as e:
+            if e.response.status_code == 407: # Proxy Authentication Required
+                handle_proxy_407(url, session)
+                # Try again
+                return download(url, dst_path, session=session, md5=md5, urlstxt=urlstxt)
             msg = "HTTPError: %s: %s\n" % (e, url)
             log.debug(msg)
             raise RuntimeError(msg)
 
+        except requests.exceptions.ConnectionError as e:
+            # requests isn't so nice here. For whatever reason, https gives this
+            # error and http gives the above error. Also, there is no status_code
+            # attribute here. We have to just check if it looks like 407.  See
+            # https://github.com/kennethreitz/requests/issues/2061.
+            if "407" in str(e): # Proxy Authentication Required
+                handle_proxy_407(url, session)
+                # Try again
+                return download(url, dst_path, session=session, md5=md5, urlstxt=urlstxt)
+        except IOError as e:
+            raise RuntimeError("Could not open '%s': %s" % (url, e))
+
         size = resp.headers.get('Content-Length')
         if size:
             size = int(size)


Test patch: