Repository: rucio/rucio

Issue:
"rucio upload" CLI fails if user has not WRITE privileges on the file to upload
Motivation
----------

Rucio 1.26.10

Error occurs at checksum computation
```
2021-12-02 15:51:22,902	DEBUG	rucio	Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/common/utils.py", line 245, in adler32
    with open(file, 'r+b') as f:
PermissionError: [Errno 13] Permission denied: '/sps/lsst/datasets/desc/DC2/Run2.2i/sim/y5-wfd/00992106/lsst_a_992106_R12_S00_z.fits'
Exception: FATAL - could not get Adler32 checksum of file /sps/lsst/datasets/desc/DC2/Run2.2i/sim/y5-wfd/00992106/lsst_a_992106_R12_S00_z.fits - [Errno 13] Permission denied: '/sps/lsst/datasets/desc/DC2/Run2.2i/sim/y5-wfd/00992106/lsst_a_992106_R12_S00_z.fits'
```


Modification
------------

Seems Rucio attempts to open the file in 'r+b' mode instead of 'rb' (since 1.26.8)
https://github.com/rucio/rucio/pull/4880/files/87c392af5873b32d23726191e0fc2dbc2e2db39c



Discussion:
@mlassnig isn't this the same observation Paul had as well?
yes, the file needs to be available in read-write mode to benefit from mmap.
While I don't understand why the file cannot be opened in read-write mode, we have to decide if we want to complicate the algorithm (check if available in read-write, if yes, use mmap, otherwise not), or just take mmap out?
I don't remember the performance benefits with mmap, if it is significant, we might check for `r+b` and fallback otherwise. If not significant, I think we should just do it the old way and forget about mmap.
`mmap` helps a lot in multi-user contexts, because it bypasses kernel-level IO. would be sad to see it go :-/ let me have a try beginning next week, maybe it's an easy change.
hi guys, we stumbled in the same issue upgrading to rucio 1.26.10 for Virgo. Would be good to have an official fix for this
@mlassnig I think it would be good to just go back to the old way of doing the checksum validation. I believe this will make us problems on the ALRB as well as soon as we deploy 1.27, at least until you can implement the dual solution with fallback.

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -271,7 +271,9 @@ def set_checksum_value(file, checksum_names_list):
 
 def adler32(file):
     """
-    An Adler-32 checksum is obtained by calculating two 16-bit checksums A and B and concatenating their bits into a 32-bit integer. A is the sum of all bytes in the stream plus one, and B is the sum of the individual values of A from each step.
+    An Adler-32 checksum is obtained by calculating two 16-bit checksums A and B
+    and concatenating their bits into a 32-bit integer. A is the sum of all bytes in the
+    stream plus one, and B is the sum of the individual values of A from each step.
 
     :param file: file name
     :returns: Hexified string, padded to 8 values.
@@ -280,15 +282,29 @@ def adler32(file):
     # adler starting value is _not_ 0
     adler = 1
 
+    can_mmap = False
     try:
         with open(file, 'r+b') as f:
-            # memory map the file
-            m = mmap.mmap(f.fileno(), 0)
-            # partial block reads at slightly increased buffer sizes
-            for block in iter(partial(m.read, io.DEFAULT_BUFFER_SIZE), b''):
-                adler = zlib.adler32(block, adler)
+            can_mmap = True
+    except:
+        pass
+
+    try:
+        # use mmap if possible
+        if can_mmap:
+            with open(file, 'r+b') as f:
+                m = mmap.mmap(f.fileno(), 0)
+                # partial block reads at slightly increased buffer sizes
+                for block in iter(partial(m.read, io.DEFAULT_BUFFER_SIZE * 8), b''):
+                    adler = zlib.adler32(block, adler)
+        else:
+            with open(file, 'rb') as f:
+                # partial block reads at slightly increased buffer sizes
+                for block in iter(partial(f.read, io.DEFAULT_BUFFER_SIZE * 8), b''):
+                    adler = zlib.adler32(block, adler)
+
     except Exception as e:
-        raise Exception('FATAL - could not get Adler32 checksum of file %s - %s' % (file, e))
+        raise Exception('FATAL - could not get Adler-32 checksum of file %s: %s' % (file, e))
 
     # backflip on 32bit -- can be removed once everything is fully migrated to 64bit
     if adler < 0:


Test patch:
diff --git a/lib/rucio/tests/test_utils.py b/lib/rucio/tests/test_utils.py
--- a/lib/rucio/tests/test_utils.py
+++ b/lib/rucio/tests/test_utils.py
@@ -18,6 +18,7 @@
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018-2019
 # - Martin Barisits <martin.barisits@cern.ch>, 2019
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020-2021
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2021
 
 import datetime
 import logging
@@ -62,7 +63,7 @@ def test_utils_adler32(self):
         assert match('[a-fA-F0-9]', ret) is not None
         assert ret == '198d03ff'
 
-        with pytest.raises(Exception, match='FATAL - could not get Adler32 checksum of file no_file - \\[Errno 2\\] No such file or directory: \'no_file\''):
+        with pytest.raises(Exception, match='FATAL - could not get Adler-32 checksum of file no_file: \\[Errno 2\\] No such file or directory: \'no_file\''):
             adler32('no_file')
 
     def test_parse_did_filter_string(self):