Repository: easybuilders/easybuild-framework

Issue:
downloading of large files fails with urllib.request with recent Python 3.x
In `download_file` we use `urllib.request`, which seems to throw an error in `url_fd.read()` when trying to read the whole file at once.

It can be reproduced with this script:
```python
import urllib.request

x = urllib.request.urlopen('https://developer.download.nvidia.com/compute/cuda/11.0.2/local_installers/cuda_11.0.2_450.51.05_linux.run')
x.read()
```

```
Traceback (most recent call last):
  File "./foo.py", line 5, in <module>
    x.read()
  File "/usr/lib/python3.8/http/client.py", line 467, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.8/http/client.py", line 608, in _safe_read
    data = self.fp.read(amt)
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
OverflowError: signed integer is greater than maximum
```

While `urllib` is bugged like this, we need to either read it in chunks ourselves or skip the naive `read` and combine it with the subsequent write to file via `shutil.copyfileobj`:
```python
with open('/dev/shm/out.raw', 'wb') as fh:
        shutil.copyfileobj(x, fh)
```

As a bonus, we won't be reading the whole thing into memory, possibly exhausting it.


Discussion:
This is Ubuntu 20.04 with:
```
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
```
Seems like a good idea to always do it?

How big does the file need to be to trigger this issue? Would be good to get this covered by the tests...
CUDA is 2924M so guessing wildly at 2 GiB.
2048 megabytes doesn't trigger it, but 2049 megabytes does.
I think we could change https://github.com/easybuilders/easybuild-framework/blob/f90a2b00d098bb7e337967d9b9aaaeb69743bf77/easybuild/tools/filetools.py#L245-L246 to handle file-like objects so we don't need to pass `x.read()` to it. But I'm not sure about how to uniformly handle both. But I guess this could well be worth it as similar issues apply for copying a file via `write_file`
Seeing this with `MCR`, after fixing the URLs to be `https`. 

Resolving patch:
diff --git a/easybuild/tools/filetools.py b/easybuild/tools/filetools.py
--- a/easybuild/tools/filetools.py
+++ b/easybuild/tools/filetools.py
@@ -217,7 +217,7 @@ def write_file(path, data, append=False, forced=False, backup=False, always_over
     overwrites current file contents without backup by default!
 
     :param path: location of file
-    :param data: contents to write to file
+    :param data: contents to write to file. Can be a file-like object of binary data
     :param append: append to existing file rather than overwrite
     :param forced: force actually writing file in (extended) dry run mode
     :param backup: back up existing file before overwriting or modifying it
@@ -246,15 +246,21 @@ def write_file(path, data, append=False, forced=False, backup=False, always_over
     # cfr. https://docs.python.org/3/library/functions.html#open
     mode = 'a' if append else 'w'
 
+    data_is_file_obj = hasattr(data, 'read')
+
     # special care must be taken with binary data in Python 3
-    if sys.version_info[0] >= 3 and isinstance(data, bytes):
+    if sys.version_info[0] >= 3 and (isinstance(data, bytes) or data_is_file_obj):
         mode += 'b'
 
     # note: we can't use try-except-finally, because Python 2.4 doesn't support it as a single block
     try:
         mkdir(os.path.dirname(path), parents=True)
         with open_file(path, mode) as fh:
-            fh.write(data)
+            if data_is_file_obj:
+                # if a file-like object was provided, use copyfileobj (which reads the file in chunks)
+                shutil.copyfileobj(data, fh)
+            else:
+                fh.write(data)
     except IOError as err:
         raise EasyBuildError("Failed to write to %s: %s", path, err)
 
@@ -710,7 +716,11 @@ def download_file(filename, url, path, forced=False):
                 url_fd = response.raw
                 url_fd.decode_content = True
             _log.debug('response code for given url %s: %s' % (url, status_code))
-            write_file(path, url_fd.read(), forced=forced, backup=True)
+            # note: we pass the file object to write_file rather than reading the file first,
+            # to ensure the data is read in chunks (which prevents problems in Python 3.9+);
+            # cfr. https://github.com/easybuilders/easybuild-framework/issues/3455
+            # and https://bugs.python.org/issue42853
+            write_file(path, url_fd, forced=forced, backup=True)
             _log.info("Downloaded file %s from url %s to %s" % (filename, url, path))
             downloaded = True
             url_fd.close()


Test patch:
diff --git a/test/framework/filetools.py b/test/framework/filetools.py
--- a/test/framework/filetools.py
+++ b/test/framework/filetools.py
@@ -705,6 +705,26 @@ def test_read_write_file(self):
         # test use of 'mode' in read_file
         self.assertEqual(ft.read_file(foo, mode='rb'), b'bar')
 
+    def test_write_file_obj(self):
+        """Test writing from a file-like object directly"""
+        # Write a text file
+        fp = os.path.join(self.test_prefix, 'test.txt')
+        fp_out = os.path.join(self.test_prefix, 'test_out.txt')
+        ft.write_file(fp, b'Hyphen: \xe2\x80\x93\nEuro sign: \xe2\x82\xac\na with dots: \xc3\xa4')
+
+        with ft.open_file(fp, 'rb') as fh:
+            ft.write_file(fp_out, fh)
+        self.assertEqual(ft.read_file(fp_out), ft.read_file(fp))
+
+        # Write a binary file
+        fp = os.path.join(self.test_prefix, 'test.bin')
+        fp_out = os.path.join(self.test_prefix, 'test_out.bin')
+        ft.write_file(fp, b'\x00\x01'+os.urandom(42)+b'\x02\x03')
+
+        with ft.open_file(fp, 'rb') as fh:
+            ft.write_file(fp_out, fh)
+        self.assertEqual(ft.read_file(fp_out, mode='rb'), ft.read_file(fp, mode='rb'))
+
     def test_is_binary(self):
         """Test is_binary function."""