Repository: conda/conda

Issue:
TLS does not appear to be verified
As far as I can tell conda is just using urllib2 which doesn't verify SSL at all in Python 2.x and in Python 3.x it doesn't do it by default. This means that even recipes which use a https link without a md5 hash there is a simple MITM code execution attack.

<!---
@huboard:{"order":158.5,"custom_state":""}
-->



Discussion:
How can we fix it? Do we need to add several more dependencies to conda?

How can it be enabled in Python 3? And why doesn't it do it by default?

So there are a few options here, the lowest impact but easiest to get wrong is to backport the ssl stuff from Python 3 into Python 2 and include your own root certificates. This is what pip did in version 1.3-1.4 and it was done incorrectly. 

My recommendation would be to either use urllib3 directly or to use requests. Pip went the requests route which allows the entire Python community to sort of centralize it's cross platform TLS verification.

Python 3 doesn't do it by default because it doesn't have any root certificates to verify against, I believe that Christian Heimes is hoping to fix this for 3.4 (or has already..).

OK. Hopefully SNI is not needed here, as I found that it required three additional packages and a patch to requests. 

SNI has pretty bad support outside of browsers. It is a nice to have but not a requirement. The difference between SNI and not SNI is that HTTPS urls will require a dedicated IP address per certificate (but a certificate can contain multiple host names) whereas SNI allows you to have multiple certificates attached to one IP address.

The other thing about missing SNI is that it'll "fail closed" instead of "fail open", so if someone has a repo or url that needs SNI they'll get an error message at any attempt to use that repo or url instead of it being silently insecure.

FWIW it appears that the default continiuum repos are also HTTP only which should change as well, they should be available via HTTPS and conda should default to using HTTPS for them. Otherwise I can inject arbitrary package contents to people installing from those repositories if I have a successful MITM.

I agree with @dstufft that standardising on requests is a good idea - the main reason is that the inclusion of the cert bundle addresses the fact that Python prior to 3.4 can't consistently access the system certs, and the Python Security Response Team will be keeping a close eye on that cert bundle now that it has a security impact on CPython itself (due to the pip bootstrapping in PEP 453).

@asmeurer Asked me on twitter if urllib3 was fine to solve this. I took a look and urllib3 does _not_ ship it's own certificates. I would recommend that Conda does _not_ try to get into the business of shipping certificates. You have to make sure you're on top of whenever they get updated, parsing them can be dificult to do properly etc. It's a tricky business to get correct. I would recommend if you can that you switch to requests and try to keep it updated sot hat you get any new certificates automatically.

However if you do want to use urllib3, I recommend using the certifi package to handle certs.

Let's just use requests. 

Will hot-swapping `urllib2` with `requests.packages.urllib3` work? I know requests is better API-wise, but we already have the code there for urllib, so this would again be a one-line fix. 

`requests.packages.urllib3` is equivalent to just using `urllib3` so all of the above still apples. requests just includes an unmodified copy of urllib3 as part of it's source.

OK, I'm looking at requests. What is the best way to do caching with requests? Will it be easy to port https://github.com/conda/conda/blob/master/conda/fetch.py#L71? If not, is there an easy way to do it without adding an additional dependency like https://github.com/Lukasa/httpcache or https://github.com/ionrock/cachecontrol? Or is using one of those the best way to go?

Oh hey, this is an opportune time for that question, as I'm just looking at ripping out pip's custom download cache and replace it with a generic HTTP cache. I've recently submitted a few pull requests to CacheControl and I would suggest using it if you can. It's a good library and with the PRs i've submitted I was able to "drop it" into pip's requests based code without any trouble.

However if you want to keep maintaining your existing cache, I did a quick once over of a port of that code to requests http://bpaste.net/show/Y5IJFRpQEwjZxswCPGwR/ I removed the error handling just because I'm lazy.

Pip ticket in case you're wondering: pypa/pip#1732

Thanks. That will save me a lot of time I'm sure. We only use this to cache the repo metadata. Packages themselves are cached manually, using the md5 which is stored in the repodata. 

I'll probably just work off your modification for now. A good motivation for moving to CacheControl would probably be if it ends up being faster than the current method somehow. I'm not sure if that's possible, but for me, with about 10 channels, it takes several seconds to load the repo metadata with every command. @srossross may also have ideas about this. 

There is more than this. We do need proper error handling (we don't want users to ever see a traceback from conda, unless there is a bug), and also it needs to be able to handle proxies (I'm hoping this is easy with requests), because quite a few users of conda are stuck behind corporate firewalls. 

Yea I just left out error handling because I'm lazy :)

I'm not sure if CacheControl would be faster than the current code, it'd likely mostly just be shifting logic around so you don't have to handle it yourself.

Requests does make proxies very easy! In fact by default it will respect `HTTP_PROXY` and `HTTPS_PROXY` environment variables so you get proxies for "free" though that. In pip we also have a command line flag that lets people do it that way too.http://docs.python-requests.org/en/latest/user/advanced/#proxies. Although if those corporate firewalls are ISA based I think you have to install some client or something to make it work. 

You'll most likely find your HTTP handling code shrink because requests handles a lot of that stuff out of the box.

One thing that's odd about requests is that `requests.get('http://repo.continuum.io/pkgs/free/osx-64/repodata.json.bz2')` returns a `str` rather than `bytes` in Python 3. Do you know any way to fix this?

Yeah, I'm pretty sure the real way to improve speed is to have some API on the binstar side that conda could use to get the merged repodata for several channels with one request. That's something that @srossross and I have discussed. 

Sorry, that's `resp.text`. Maybe there is some other attribute of the Response object I should be using. 

It should return a Response object, which has a `text` attribute which will be the content of the request decoded with whatever encoding it detects the text of the response is in, and a `content` attribute which is just the (mostly raw) bytes from the wire.

I say mostly raw because it'll automatically do gzip decoding even for `resp.content` if the server has the `Content-Encoding: gzip` header.

Oh, I see, it's `resp.content`. I guess it makes sense that "text" would be a str, even if it isn't really text :)

Is there a similar header for bz2?

Also if you want to do like progress bar, you can do:

``` python
resp = requests.get("https:///", stream=True)
chunk = b""
for i, chunk in enumerate(resp.iter_content(4096)):
    print("Chunk #{}".format(i))
    downloaded += chunk
```

The above will open a connection, but _not_ download the entire response body, but will instead download a 4096 byte chunk for each loop iteration.

OK, I haven't gotten to the pkg downloading code yet. That's next. We hardly need a progress bar for fetching the repodata. 

Thanks for the advice so far.

Not by default, HTTP connections typically only use deflate or gzip (both of those will be decoded automatically). Anything else won't be. Although if you wanted it handled automatically you could probably make requests do it by making your own transport adapter that passed one into the urllib3 HTTPReponse I think.

Also if you're doing multiple connections to the same host, you'll want to use a requests session.

``` python
import requests
session = requests.session()
session.get("https://....")
```

The benefit is that you'll get connection pooling and keep alive connections between HTTP requests (so you won't have to do the TLS handshake for each one, things will generally be much faster etc). You can also use requests sessions to share some settings to prevent you from having to pass them into each function call. (See http://docs.python-requests.org/en/latest/api/#sessionapi)

I didn't notice a speed difference using session, but I'll use it anyway.


Resolving patch:
diff --git a/conda/connection.py b/conda/connection.py
--- a/conda/connection.py
+++ b/conda/connection.py
@@ -7,106 +7,368 @@
 from __future__ import print_function, division, absolute_import
 
 from logging import getLogger
+import re
+import mimetypes
+import os
+import email
+import base64
+import ftplib
+import cgi
+from io import BytesIO
 
-from conda.compat import PY3, string_types
-from conda.compat import iteritems, input
+from conda.compat import urlparse, StringIO
 from conda.config import get_proxy_servers
 
-if PY3:
-    # Python 3.x
-    import urllib.request as urllib2
-    from urllib import parse as urlparse
-else:
-    # Python 2.x
-    import urllib2
-    import urlparse
+import requests
 
+RETRIES = 3
 
 log = getLogger(__name__)
 
-# 1. get proxies if needed. a proxy for each  protocol
-# 2. handle authentication
-# basic, digest, and nltm (windows) authentications should be handled.
-# 3. handle any protocol
-# typically http, https, ftp
-
-# 1. get the proxies list
-# urllib can only get proxies on windows and mac. so on linux or if the user
-# wants to specify the proxy there has to be a way to do that. TODO get proxies
-# from condarc and overrwrite any system proxies
-# the proxies are in a dict {'http':'http://proxy:8080'}
-# protocol:proxyserver
-proxies_dict = get_proxy_servers() or urllib2.getproxies()
-
-#2. handle authentication
-
-proxypwdmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
-
-
-def get_userandpass(proxytype='', realm=''):
-    """a function to get username and password from terminal.
-    can be replaced with anything like some gui"""
-    import getpass
-
-    uname = input(proxytype + ' proxy username:')
-    pword = getpass.getpass()
-    return uname, pword
-
-
-# a procedure that needs to be executed with changes to handlers
-def installopener():
-    opener = urllib2.build_opener(
-        urllib2.ProxyHandler(proxies_dict),
-        urllib2.ProxyBasicAuthHandler(proxypwdmgr),
-        urllib2.ProxyDigestAuthHandler(proxypwdmgr),
-        urllib2.HTTPHandler,
-    )
-    # digest auth may not work with all proxies
-    # http://bugs.python.org/issue16095
-    # could add windows/nltm authentication here
-    #opener=urllib2.build_opener(urllib2.ProxyHandler(proxies_dict), urllib2.HTTPHandler)
-
-    urllib2.install_opener(opener)
-
-
-firstconnection = True
-#i made this func so i wouldn't alter the original code much
-def connectionhandled_urlopen(request):
-    """handles aspects of establishing the connection with the remote"""
-
-    installopener()
-
-    if isinstance(request, string_types):
-        request = urllib2.Request(request)
-
-    try:
-        return urllib2.urlopen(request)
-
-    except urllib2.HTTPError as HTTPErrorinst:
-        if HTTPErrorinst.code in (407, 401):
-            # proxy authentication error
-            # ...(need to auth) or supplied creds failed
-            if HTTPErrorinst.code == 401:
-                log.debug('proxy authentication failed')
-            #authenticate and retry
-            uname, pword = get_userandpass()
-            #assign same user+pwd to all protocols (a reasonable assumption) to
-            #decrease user input. otherwise you'd need to assign a user/pwd to
-            #each proxy type
-            if firstconnection == True:
-                for aprotocol, aproxy in iteritems(proxies_dict):
-                    proxypwdmgr.add_password(None, aproxy, uname, pword)
-                firstconnection == False
-            else: #...assign a uname pwd for the specific protocol proxy type
-                assert(firstconnection == False)
-                protocol = urlparse.urlparse(request.get_full_url()).scheme
-                proxypwdmgr.add_password(None, proxies_dict[protocol],
-                                         uname, pword)
-            installopener()
-            # i'm uncomfortable with this
-            # but i just want to exec to start from the top again
-            return connectionhandled_urlopen(request)
-        raise
-
-    except:
-        raise
+# Modified from code in pip/download.py:
+
+# Copyright (c) 2008-2014 The pip developers (see AUTHORS.txt file)
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+class CondaSession(requests.Session):
+
+    timeout = None
+
+    def __init__(self, *args, **kwargs):
+        retries = kwargs.pop('retries', RETRIES)
+
+        super(CondaSession, self).__init__(*args, **kwargs)
+
+        self.proxies = get_proxy_servers()
+
+        # Configure retries
+        if retries:
+            http_adapter = requests.adapters.HTTPAdapter(max_retries=retries)
+            self.mount("http://", http_adapter)
+            self.mount("https://", http_adapter)
+
+        # Enable file:// urls
+        self.mount("file://", LocalFSAdapter())
+
+        # Enable ftp:// urls
+        self.mount("ftp://", FTPAdapter())
+
+class LocalFSAdapter(requests.adapters.BaseAdapter):
+
+    def send(self, request, stream=None, timeout=None, verify=None, cert=None,
+             proxies=None):
+        pathname = url_to_path(request.url)
+
+        resp = requests.models.Response()
+        resp.status_code = 200
+        resp.url = request.url
+
+        try:
+            stats = os.stat(pathname)
+        except OSError as exc:
+            resp.status_code = 404
+            resp.raw = exc
+        else:
+            modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
+            content_type = mimetypes.guess_type(pathname)[0] or "text/plain"
+            resp.headers = requests.structures.CaseInsensitiveDict({
+                "Content-Type": content_type,
+                "Content-Length": stats.st_size,
+                "Last-Modified": modified,
+            })
+
+            resp.raw = open(pathname, "rb")
+            resp.close = resp.raw.close
+
+        return resp
+
+    def close(self):
+        pass
+
+def url_to_path(url):
+    """
+    Convert a file: URL to a path.
+    """
+    assert url.startswith('file:'), (
+        "You can only turn file: urls into filenames (not %r)" % url)
+    path = url[len('file:'):].lstrip('/')
+    path = urlparse.unquote(path)
+    if _url_drive_re.match(path):
+        path = path[0] + ':' + path[2:]
+    else:
+        path = '/' + path
+    return path
+
+_url_drive_re = re.compile('^([a-z])[:|]', re.I)
+
+# Taken from requests-ftp
+# (https://github.com/Lukasa/requests-ftp/blob/master/requests_ftp/ftp.py)
+
+# Copyright 2012 Cory Benfield
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+class FTPAdapter(requests.adapters.BaseAdapter):
+    '''A Requests Transport Adapter that handles FTP urls.'''
+    def __init__(self):
+        super(FTPAdapter, self).__init__()
+
+        # Build a dictionary keyed off the methods we support in upper case.
+        # The values of this dictionary should be the functions we use to
+        # send the specific queries.
+        self.func_table = {'LIST': self.list,
+                           'RETR': self.retr,
+                           'STOR': self.stor,
+                           'NLST': self.nlst,
+                           'GET':  self.retr,}
+
+    def send(self, request, **kwargs):
+        '''Sends a PreparedRequest object over FTP. Returns a response object.
+        '''
+        # Get the authentication from the prepared request, if any.
+        auth = self.get_username_password_from_header(request)
+
+        # Next, get the host and the path.
+        host, port, path = self.get_host_and_path_from_url(request)
+
+        # Sort out the timeout.
+        timeout = kwargs.get('timeout', None)
+
+        # Establish the connection and login if needed.
+        self.conn = ftplib.FTP()
+        self.conn.connect(host, port, timeout)
+
+        if auth is not None:
+            self.conn.login(auth[0], auth[1])
+        else:
+            self.conn.login()
+
+        # Get the method and attempt to find the function to call.
+        resp = self.func_table[request.method](path, request)
+
+        # Return the response.
+        return resp
+
+    def close(self):
+        '''Dispose of any internal state.'''
+        # Currently this is a no-op.
+        pass
+
+    def list(self, path, request):
+        '''Executes the FTP LIST command on the given path.'''
+        data = StringIO()
+
+        # To ensure the StringIO gets cleaned up, we need to alias its close
+        # method to the release_conn() method. This is a dirty hack, but there
+        # you go.
+        data.release_conn = data.close
+
+        self.conn.cwd(path)
+        code = self.conn.retrbinary('LIST', data_callback_factory(data))
+
+        # When that call has finished executing, we'll have all our data.
+        response = build_text_response(request, data, code)
+
+        # Close the connection.
+        self.conn.close()
+
+        return response
+
+    def retr(self, path, request):
+        '''Executes the FTP RETR command on the given path.'''
+        data = BytesIO()
+
+        # To ensure the BytesIO gets cleaned up, we need to alias its close
+        # method. See self.list().
+        data.release_conn = data.close
+
+        code = self.conn.retrbinary('RETR ' + path, data_callback_factory(data))
+
+        response = build_binary_response(request, data, code)
+
+        # Close the connection.
+        self.conn.close()
+
+        return response
+
+    def stor(self, path, request):
+        '''Executes the FTP STOR command on the given path.'''
+
+        # First, get the file handle. We assume (bravely)
+        # that there is only one file to be sent to a given URL. We also
+        # assume that the filename is sent as part of the URL, not as part of
+        # the files argument. Both of these assumptions are rarely correct,
+        # but they are easy.
+        data = parse_multipart_files(request)
+
+        # Split into the path and the filename.
+        path, filename = os.path.split(path)
+
+        # Switch directories and upload the data.
+        self.conn.cwd(path)
+        code = self.conn.storbinary('STOR ' + filename, data)
+
+        # Close the connection and build the response.
+        self.conn.close()
+
+        response = build_binary_response(request, BytesIO(), code)
+
+        return response
+
+    def nlst(self, path, request):
+        '''Executes the FTP NLST command on the given path.'''
+        data = StringIO()
+
+        # Alias the close method.
+        data.release_conn = data.close
+
+        self.conn.cwd(path)
+        code = self.conn.retrbinary('NLST', data_callback_factory(data))
+
+        # When that call has finished executing, we'll have all our data.
+        response = build_text_response(request, data, code)
+
+        # Close the connection.
+        self.conn.close()
+
+        return response
+
+    def get_username_password_from_header(self, request):
+        '''Given a PreparedRequest object, reverse the process of adding HTTP
+        Basic auth to obtain the username and password. Allows the FTP adapter
+        to piggyback on the basic auth notation without changing the control
+        flow.'''
+        auth_header = request.headers.get('Authorization')
+
+        if auth_header:
+            # The basic auth header is of the form 'Basic xyz'. We want the
+            # second part. Check that we have the right kind of auth though.
+            encoded_components = auth_header.split()[:2]
+            if encoded_components[0] != 'Basic':
+                raise AuthError('Invalid form of Authentication used.')
+            else:
+                encoded = encoded_components[1]
+
+            # Decode the base64 encoded string.
+            decoded = base64.b64decode(encoded)
+
+            # The string is of the form 'username:password'. Split on the
+            # colon.
+            components = decoded.split(':')
+            username = components[0]
+            password = components[1]
+            return (username, password)
+        else:
+            # No auth header. Return None.
+            return None
+
+    def get_host_and_path_from_url(self, request):
+        '''Given a PreparedRequest object, split the URL in such a manner as to
+        determine the host and the path. This is a separate method to wrap some
+        of urlparse's craziness.'''
+        url = request.url
+        # scheme, netloc, path, params, query, fragment = urlparse(url)
+        parsed = urlparse.urlparse(url)
+        path = parsed.path
+
+        # If there is a slash on the front of the path, chuck it.
+        if path[0] == '/':
+            path = path[1:]
+
+        host = parsed.hostname
+        port = parsed.port or 0
+
+        return (host, port, path)
+
+def data_callback_factory(variable):
+    '''Returns a callback suitable for use by the FTP library. This callback
+    will repeatedly save data into the variable provided to this function. This
+    variable should be a file-like structure.'''
+    def callback(data):
+        variable.write(data)
+        return
+
+    return callback
+
+class AuthError(Exception):
+    '''Denotes an error with authentication.'''
+    pass
+
+def build_text_response(request, data, code):
+    '''Build a response for textual data.'''
+    return build_response(request, data, code, 'ascii')
+
+def build_binary_response(request, data, code):
+    '''Build a response for data whose encoding is unknown.'''
+    return build_response(request, data, code,  None)
+
+def build_response(request, data, code, encoding):
+    '''Builds a response object from the data returned by ftplib, using the
+    specified encoding.'''
+    response = requests.Response()
+
+    response.encoding = encoding
+
+    # Fill in some useful fields.
+    response.raw = data
+    response.url = request.url
+    response.request = request
+    response.status_code = code.split()[0]
+
+    # Make sure to seek the file-like raw object back to the start.
+    response.raw.seek(0)
+
+    # Run the response hook.
+    response = requests.hooks.dispatch_hook('response', request.hooks, response)
+    return response
+
+def parse_multipart_files(request):
+    '''Given a prepared reqest, return a file-like object containing the
+    original data. This is pretty hacky.'''
+    # Start by grabbing the pdict.
+    _, pdict = cgi.parse_header(request.headers['Content-Type'])
+
+    # Now, wrap the multipart data in a BytesIO buffer. This is annoying.
+    buf = BytesIO()
+    buf.write(request.body)
+    buf.seek(0)
+
+    # Parse the data. Simply take the first file.
+    data = cgi.parse_multipart(buf, pdict)
+    _, filedata = data.popitem()
+    buf.close()
+
+    # Get a BytesIO now, and write the file into it.
+    buf = BytesIO()
+    buf.write(''.join(filedata))
+    buf.seek(0)
+
+    return buf
diff --git a/conda/fetch.py b/conda/fetch.py
--- a/conda/fetch.py
+++ b/conda/fetch.py
@@ -8,7 +8,6 @@
 
 import os
 import bz2
-import sys
 import json
 import shutil
 import hashlib
@@ -18,22 +17,18 @@
 
 from conda import config
 from conda.utils import memoized
-from conda.connection import connectionhandled_urlopen
-from conda.compat import PY3, itervalues, get_http_value
+from conda.connection import CondaSession
+from conda.compat import itervalues, get_http_value
 from conda.lock import Locked
 
-if PY3:
-    import urllib.request as urllib2
-else:
-    import urllib2
-
+import requests
 
 log = getLogger(__name__)
 dotlog = getLogger('dotupdate')
 stdoutlog = getLogger('stdoutlog')
+stderrlog = getLogger('stderrlog')
 
 fail_unknown_host = False
-retries = 3
 
 
 def create_cache_dir():
@@ -55,9 +50,11 @@ def add_http_value_to_dict(u, http_key, d, dict_key):
         d[dict_key] = value
 
 
-def fetch_repodata(url, cache_dir=None, use_cache=False):
+def fetch_repodata(url, cache_dir=None, use_cache=False, session=None):
     dotlog.debug("fetching repodata: %s ..." % url)
 
+    session = session or CondaSession()
+
     cache_path = join(cache_dir or create_cache_dir(), cache_fn_url(url))
     try:
         cache = json.load(open(cache_path))
@@ -67,33 +64,32 @@ def fetch_repodata(url, cache_dir=None, use_cache=False):
     if use_cache:
         return cache
 
-    request = urllib2.Request(url + 'repodata.json.bz2')
-    if '_etag' in cache:
-        request.add_header('If-None-Match', cache['_etag'])
-    if '_mod' in cache:
-        request.add_header('If-Modified-Since', cache['_mod'])
+    headers = {}
+    if "_tag" in cache:
+        headers["If-None-Match"] = cache["_etag"]
+    if "_mod" in cache:
+        headers["If-Modified-Since"] = cache["_mod"]
 
     try:
-        u = connectionhandled_urlopen(request)
-        data = u.read()
-        u.close()
-        cache = json.loads(bz2.decompress(data).decode('utf-8'))
-        add_http_value_to_dict(u, 'Etag', cache, '_etag')
-        add_http_value_to_dict(u, 'Last-Modified', cache, '_mod')
+        resp = session.get(url + 'repodata.json.bz2', headers=headers)
+        resp.raise_for_status()
+        if resp.status_code != 304:
+            cache = json.loads(bz2.decompress(resp.content).decode('utf-8'))
 
     except ValueError:
         raise RuntimeError("Invalid index file: %srepodata.json.bz2" % url)
 
-    except urllib2.HTTPError as e:
-        msg = "HTTPError: %d  %s  %s\n" % (e.code, e.msg, url)
+    except requests.exceptions.HTTPError as e:
+        msg = "HTTPError: %s: %s\n" % (e, url)
         log.debug(msg)
-        if e.code != 304:
-            raise RuntimeError(msg)
+        raise RuntimeError(msg)
 
-    except urllib2.URLError as e:
-        sys.stderr.write("Error: unknown host: %s (%r)\n" % (url, e))
+    except requests.exceptions.ConnectionError as e:
+        msg = "Connection error: %s: %s\n" % (e, url)
+        stderrlog.info('Could not connect to %s\n' % url)
+        log.debug(msg)
         if fail_unknown_host:
-            sys.exit(1)
+            raise RuntimeError(msg)
 
     cache['_url'] = url
     try:
@@ -104,16 +100,16 @@ def fetch_repodata(url, cache_dir=None, use_cache=False):
 
     return cache or None
 
-
 @memoized
 def fetch_index(channel_urls, use_cache=False, unknown=False):
     log.debug('channel_urls=' + repr(channel_urls))
     index = {}
     stdoutlog.info("Fetching package metadata: ")
+    session = CondaSession()
     for url in reversed(channel_urls):
         if config.allowed_channels and url not in config.allowed_channels:
             sys.exit("\nError: URL '%s' not in allowed channels" % url)
-        repodata = fetch_repodata(url, use_cache=use_cache)
+        repodata = fetch_repodata(url, use_cache=use_cache, session=session)
         if repodata is None:
             continue
         new_index = repodata['packages']
@@ -141,107 +137,80 @@ def fetch_index(channel_urls, use_cache=False, unknown=False):
 
     return index
 
-
-def fetch_pkg(info, dst_dir=None):
+def fetch_pkg(info, dst_dir=None, session=None):
     '''
     fetch a package given by `info` and store it into `dst_dir`
     '''
     if dst_dir is None:
         dst_dir = config.pkgs_dirs[0]
 
+    session = session or CondaSession()
+
     fn = '%(name)s-%(version)s-%(build)s.tar.bz2' % info
     url = info['channel'] + fn
     log.debug("url=%r" % url)
     path = join(dst_dir, fn)
-    pp = path + '.part'
+
+    download(url, path, session=session, md5=info['md5'], urlstxt=True)
+
+def download(url, dst_path, session=None, md5=None, urlstxt=False):
+    pp = dst_path + '.part'
+    dst_dir = os.path.split(dst_path)[0]
+    session = session or CondaSession()
 
     with Locked(dst_dir):
-        for x in range(retries):
-            try:
-                fi = connectionhandled_urlopen(url)
-            except IOError:
-                log.debug("attempt %d failed at urlopen" % x)
-                continue
-            if fi is None:
-                log.debug("could not fetch (urlopen returned None)")
-                continue
-            n = 0
-            h = hashlib.new('md5')
-            getLogger('fetch.start').info((fn, info['size']))
-            need_retry = False
-            try:
-                fo = open(pp, 'wb')
-            except IOError:
-                raise RuntimeError("Could not open %r for writing.  "
-                          "Permissions problem or missing directory?" % pp)
-            while True:
-                try:
-                    chunk = fi.read(16384)
-                except IOError:
-                    need_retry = True
-                    break
-                if not chunk:
-                    break
-                try:
-                    fo.write(chunk)
-                except IOError:
-                    raise RuntimeError("Failed to write to %r." % pp)
-                h.update(chunk)
-                n += len(chunk)
-                getLogger('fetch.update').info(n)
+        try:
+            resp = session.get(url, stream=True)
+        except IOError:
+            raise RuntimeError("Could not open '%s'" % url)
+        except requests.exceptions.HTTPError as e:
+            msg = "HTTPError: %s: %s\n" % (e, url)
+            log.debug(msg)
+            raise RuntimeError(msg)
 
-            fo.close()
-            if need_retry:
-                continue
+        size = resp.headers.get('Content-Length')
+        if size:
+            size = int(size)
+            fn = basename(dst_path)
+            getLogger('fetch.start').info((fn[:14], size))
+
+        n = 0
+        if md5:
+            h = hashlib.new('md5')
+        try:
+            with open(pp, 'wb') as fo:
+                for chunk in resp.iter_content(2**14):
+                    try:
+                        fo.write(chunk)
+                    except IOError:
+                        raise RuntimeError("Failed to write to %r." % pp)
+                    if md5:
+                        h.update(chunk)
+                    n += len(chunk)
+                    if size:
+                        getLogger('fetch.update').info(n)
+        except IOError:
+            raise RuntimeError("Could not open %r for writing.  "
+                "Permissions problem or missing directory?" % pp)
 
-            fi.close()
+        if size:
             getLogger('fetch.stop').info(None)
-            if h.hexdigest() != info['md5']:
-                raise RuntimeError("MD5 sums mismatch for download: %s (%s != %s)" % (fn, h.hexdigest(), info['md5']))
-            try:
-                os.rename(pp, path)
-            except OSError:
-                raise RuntimeError("Could not rename %r to %r." % (pp, path))
+
+        if md5 and h.hexdigest() != md5:
+            raise RuntimeError("MD5 sums mismatch for download: %s (%s != %s)" % (url, h.hexdigest(), md5))
+
+        try:
+            os.rename(pp, dst_path)
+        except OSError as e:
+            raise RuntimeError("Could not rename %r to %r: %r" % (pp,
+                dst_path, e))
+
+        if urlstxt:
             try:
                 with open(join(dst_dir, 'urls.txt'), 'a') as fa:
                     fa.write('%s\n' % url)
             except IOError:
                 pass
-            return
-
-    raise RuntimeError("Could not locate '%s'" % url)
-
-
-def download(url, dst_path):
-    try:
-        u = connectionhandled_urlopen(url)
-    except IOError:
-        raise RuntimeError("Could not open '%s'" % url)
-    except ValueError as e:
-        raise RuntimeError(e)
-
-    size = get_http_value(u, 'Content-Length')
-    if size:
-        size = int(size)
-        fn = basename(dst_path)
-        getLogger('fetch.start').info((fn[:14], size))
-
-    n = 0
-    fo = open(dst_path, 'wb')
-    while True:
-        chunk = u.read(16384)
-        if not chunk:
-            break
-        fo.write(chunk)
-        n += len(chunk)
-        if size:
-            getLogger('fetch.update').info(n)
-    fo.close()
-
-    u.close()
-    if size:
-        getLogger('fetch.stop').info(None)
-
 
 class TmpDownload(object):
     """


Test patch: