Repository: easybuilders/easybuild-framework

Issue:
CentOS 6 / Python 2.6 support
Hello,

Im' trying to use easybuild on CentOS 6, and it looks like I've got some problem regarding https source download using SNI. Maybe it is a known issue for some time, or I've missed something in the documentation. Sorry if that's the case.  

From my understanding, the problem is around the ssl implementation. For example:

`
$ eb --debug hwloc-1.6.2-GCC-4.7.2.eb
[....]
== 2018-06-21 09:41:53,061 filetools.py:488 INFO Attempt 1 of downloading http://www.open-mpi.org/software/hwloc/v1.6/downloads/hwloc-1.6.2.tar.gz to /soft/adm/easybuild/sources/h/hwloc/hwloc-1.6.2.tar.gz failed, trying again...
== 2018-06-21 09:41:53,894 filetools.py:482 WARNING IOError occurred while trying to download http://www.open-mpi.org/software/hwloc/v1.6/downloads/hwloc-1.6.2.tar.gz to /soft/adm/easybuild/sources/h/hwloc/hwloc-1.6.2.tar.gz: <urlopen error [Errno 1] _ssl.c:492: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
[....]
`

Here is a simple reproducer:
```
#!/usr/bin/env python
# generates urllib2.URLError: <urlopen error [Errno 1] _ssl.c:492: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
# on CentOS 6.9 with python 2.6.6
import urllib2

url = "https://www.open-mpi.org/software/hwloc/v1.6/downloads/hwloc-1.6.2.tar.gz"
url_req = urllib2.Request(url, headers={'User-Agent': 'EasyBuild',  "Accept" : "*/*"})
urllib2.urlopen(url_req)
```

(https://gist.github.com/jbd/4db3e4c4b2eb74c466b9fc2585b1afa2)

Is there a way around this in the context of EasyBuild ? 

Further details below.

For example, 
```
$ cat /etc/centos-release 
CentOS release 6.9 (Final)
$ python --version
Python 2.6.6
``` 

```
$ eb --version
This is EasyBuild 3.6.1 (framework: 3.6.1, easyblocks: 3.6.1) on host brucciu.itcrowd
```

```
$ eb --show-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F: configuration file)
#
buildpath      (E) = /soft/adm/easybuild/build
configfiles    (E) = /soft/adm/easybuild/conf/config.cfg
containerpath  (E) = /soft/adm/easybuild/containers
installpath    (E) = /soft/adm/easybuild
packagepath    (E) = /soft/adm/easybuild/packages
prefix         (E) = /soft/adm/easybuild
repositorypath (E) = /soft/adm/easybuild/ebfiles_repo
robot-paths    (D) = /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs
sourcepath     (E) = /soft/adm/easybuild/sources
```

```
$ eb --debug hwloc-1.6.2-GCC-4.7.2.eb
== temporary log file in case of crash /home/eb/eb-zqnIyY/easybuild-QfVm7N.log
== processing EasyBuild easyconfig /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/hwloc-1.6.2-GCC-4.7.2.eb
== building and installing hwloc/1.6.2-GCC-4.7.2...
== fetching files...
== FAILED: Installation ended unsuccessfully (build directory: /soft/adm/easybuild/build/hwloc/1.6.2/GCC-4.7.2): build failed (first 300 chars): Couldn't find file hwloc-1.6.2.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/h/hwloc/hwloc-1.6.2.tar.gz, /soft/adm/easy
== Results of the build can be found in the log file(s) /home/eb/eb-zqnIyY/easybuild-hwloc-1.6.2-20180621.094152.PxciI.log
ERROR: Build of /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/hwloc-1.6.2-GCC-4.7.2.eb failed (err: "build failed (first 300 chars): Couldn't find file hwloc-1.6.2.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/h/hwloc/hwloc-1.6.2.tar.gz, /soft/adm/easy")
```

Output log: https://gist.github.com/jbd/7e0a07b9b2c8295989d65bca5ababd51




Discussion:
@jbd Can you check whether `export PYTHONHTTPSVERIFY=0` (*exactly* that) fixes your problem?

See for https://www.python.org/dev/peps/pep-0493/ (extensive) context...

This is a breaking change in Python 2.7.12, but it may have been backported to the Python 2.6.x in CentOS 6...
@boegel Thank you for the pep link.

Looks the same with `export PYTHONHTTPSVERIFY=0`:

```
$ export PYTHONHTTPSVERIFY=0
$ eb --debug hwloc-1.6.2-GCC-4.7.2.eb
== temporary log file in case of crash /home/eb/eb-Cvdx4V/easybuild-C8jsEA.log
== processing EasyBuild easyconfig /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/hwloc-1.6.2-GCC-4.7.2.eb
== building and installing hwloc/1.6.2-GCC-4.7.2...
== fetching files...
== FAILED: Installation ended unsuccessfully (build directory: /soft/adm/easybuild/build/hwloc/1.6.2/GCC-4.7.2): build failed (first 300 chars): Couldn't find file hwloc-1.6.2.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/h/hwloc/hwloc-1.6.2.tar.gz, /soft/adm/easy
== Results of the build can be found in the log file(s) /home/eb/eb-Cvdx4V/easybuild-hwloc-1.6.2-20180621.134320.Jywti.log
ERROR: Build of /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/hwloc-1.6.2-GCC-4.7.2.eb failed (err: "build failed (first 300 chars): Couldn't find file hwloc-1.6.2.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order): /soft/adm/easybuild/software/EasyBuild/3.6.1/lib/python2.6/site-packages/easybuild_easyconfigs-3.6.1-py2.6.egg/easybuild/easyconfigs/h/hwloc/h/hwloc/hwloc-1.6.2.tar.gz, /soft/adm/easy")
```

Full output: https://gist.github.com/jbd/7fa7d99b9be45d4a6a6fea11a3bb2e3a

I'll try to find if the CentOS 6 python 2.6 include the change. 
Update: a `grep -R PYTHONHTTPSVERIFY` inside the last python CentOS 6.9 SRPM (http://vault.centos.org/6.9/os/Source/SPackages/python-2.6.6-66.el6_8.src.rpm) returns nothing.
Just an update to give additional info:
```
$ rpm -qa|grep python
python-setuptools-0.6.10-4el6_9.noarch
python-devel-2.6.6-66.el6_8.x86_64
python-libs-2.6.6-66.el6_8.x86_64
python-urlgrabber-3.9.1-11.el6.noarch
libproxy-python-0.3.0-10.el6.x86_64
python-virtualenv-12.0.7-1.el6.noarch
python-2.6.6-66.el6_8.x86_64
rpm-python-4.8.0-55.el6.x86_64
python-pycurl-7.19.0-9.el6.x86_64
python-iniparse-0.3.1-2.1.el6.noarch
```

I'm using a CentOS 6 singularity container to perform my tests (as mentioned in https://github.com/easybuilders/easybuild-framework/issues/2389#issuecomment-398760388). The easybuild bootstrap has been done using the python from a virtualenv (the handshake problem remains the same).

I'll try to reproduce it on a vanilla CentOS 6.9 up-to-date installation.

Resolving patch:
diff --git a/easybuild/tools/filetools.py b/easybuild/tools/filetools.py
--- a/easybuild/tools/filetools.py
+++ b/easybuild/tools/filetools.py
@@ -60,6 +60,11 @@
 from easybuild.tools.config import build_option
 from easybuild.tools import run
 
+try:
+    import requests
+    HAVE_REQUESTS = True
+except ImportError:
+    HAVE_REQUESTS = False
 
 _log = fancylogger.getLogger('filetools', fname=False)
 
@@ -460,26 +465,47 @@ def download_file(filename, url, path, forced=False):
     attempt_cnt = 0
 
     # use custom HTTP header
-    url_req = urllib2.Request(url, headers={'User-Agent': 'EasyBuild',  "Accept" : "*/*"})
+    headers = {'User-Agent': 'EasyBuild', 'Accept': '*/*'}
+    # for backward compatibility, and to avoid relying on 3rd party Python library 'requests'
+    url_req = urllib2.Request(url, headers=headers)
+    used_urllib = urllib2
 
     while not downloaded and attempt_cnt < max_attempts:
         try:
-            # urllib2 does the right thing for http proxy setups, urllib does not!
-            url_fd = urllib2.urlopen(url_req, timeout=timeout)
-            _log.debug('response code for given url %s: %s' % (url, url_fd.getcode()))
+            if used_urllib is urllib2:
+                # urllib2 does the right thing for http proxy setups, urllib does not!
+                url_fd = urllib2.urlopen(url_req, timeout=timeout)
+                status_code = url_fd.getcode()
+            else:
+                response = requests.get(url, headers=headers, stream=True, timeout=timeout)
+                status_code = response.status_code
+                response.raise_for_status()
+                url_fd = response.raw
+                url_fd.decode_content = True
+            _log.debug('response code for given url %s: %s' % (url, status_code))
             write_file(path, url_fd.read(), forced=forced, backup=True)
             _log.info("Downloaded file %s from url %s to %s" % (filename, url, path))
             downloaded = True
             url_fd.close()
-        except urllib2.HTTPError as err:
-            if 400 <= err.code <= 499:
-                _log.warning("URL %s was not found (HTTP response code %s), not trying again" % (url, err.code))
+        except used_urllib.HTTPError as err:
+            if used_urllib is urllib2:
+                status_code = err.code
+            if 400 <= status_code <= 499:
+                _log.warning("URL %s was not found (HTTP response code %s), not trying again" % (url, status_code))
                 break
             else:
                 _log.warning("HTTPError occurred while trying to download %s to %s: %s" % (url, path, err))
                 attempt_cnt += 1
         except IOError as err:
             _log.warning("IOError occurred while trying to download %s to %s: %s" % (url, path, err))
+            error_re = re.compile(r"<urlopen error \[Errno 1\] _ssl.c:.*: error:.*:"
+                                  "SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>")
+            if error_re.match(str(err)):
+                if not HAVE_REQUESTS:
+                    raise EasyBuildError("SSL issues with urllib2. If you are using RHEL/CentOS 6.x please "
+                                         "install the python-requests and pyOpenSSL RPM packages and try again.")
+                _log.info("Downloading using requests package instead of urllib2")
+                used_urllib = requests
             attempt_cnt += 1
         except Exception, err:
             raise EasyBuildError("Unexpected error occurred when trying to download %s to %s: %s", url, path, err)


Test patch:
diff --git a/test/framework/filetools.py b/test/framework/filetools.py
--- a/test/framework/filetools.py
+++ b/test/framework/filetools.py
@@ -59,6 +59,18 @@ class FileToolsTest(EnhancedTestCase):
         ('0_foo+0x0x#-$__', 'EB_0_underscore_foo_plus_0x0x_hash__minus__dollar__underscore__underscore_'),
     ]
 
+    def setUp(self):
+        """Test setup."""
+        super(FileToolsTest, self).setUp()
+
+        self.orig_filetools_urllib2_urlopen = ft.urllib2.urlopen
+
+    def tearDown(self):
+        """Cleanup."""
+        super(FileToolsTest, self).tearDown()
+
+        ft.urllib2.urlopen = self.orig_filetools_urllib2_urlopen
+
     def test_extract_cmd(self):
         """Test various extract commands."""
         tests = [
@@ -317,7 +329,7 @@ def test_download_file(self):
         opts = init_config(args=['--download-timeout=5.3'])
         init_config(build_options={'download_timeout': opts.download_timeout})
         target_location = os.path.join(self.test_prefix, 'jenkins_robots.txt')
-        url = 'https://jenkins1.ugent.be/robots.txt'
+        url = 'https://raw.githubusercontent.com/easybuilders/easybuild-framework/master/README.rst'
         try:
             urllib2.urlopen(url)
             res = ft.download_file(fn, url, target_location)
@@ -349,6 +361,30 @@ def test_download_file(self):
         self.assertTrue(os.path.exists(target_location))
         self.assertTrue(os.path.samefile(path, target_location))
 
+    def test_download_file_requests_fallback(self):
+        """Test fallback to requests in download_file function."""
+        url = 'https://raw.githubusercontent.com/easybuilders/easybuild-framework/master/README.rst'
+        fn = 'README.rst'
+        target = os.path.join(self.test_prefix, fn)
+
+        # replace urllib2.urlopen with function that raises SSL error
+        def fake_urllib2_open(*args, **kwargs):
+            error_msg = "<urlopen error [Errno 1] _ssl.c:510: error:12345:"
+            error_msg += "SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>"
+            raise IOError(error_msg)
+
+        ft.urllib2.urlopen = fake_urllib2_open
+
+        # if requests is available, file is downloaded
+        if ft.HAVE_REQUESTS:
+            res = ft.download_file(fn, url, target)
+            self.assertTrue(res and os.path.exists(res))
+            self.assertTrue("https://easybuilders.github.io/easybuild" in ft.read_file(res))
+
+        # without requests being available, error is raised
+        ft.HAVE_REQUESTS = False
+        self.assertErrorRegex(EasyBuildError, "SSL issues with urllib2", ft.download_file, fn, url, target)
+
     def test_mkdir(self):
         """Test mkdir function."""