Repository: rucio/rucio
Issue: STORM-Rucio checksum mismatch when adler32 has leading `0`(s) #3494 #3538

Motivation
----------
During `rucio upload` towards a STORM RSE, a checksum mismatch occurs when `adler32` of the file has leading `0`(s).

```
2020-11-05 17:02:38,917    18186    INFO    Successful upload of temporary file. davs://xfer.cr.cnaf.infn.it:8443/escape/rucio/cnaf_storm/ESCAPE_CERN_TEAM-noise/53/5c/auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K.rucio.upload
2020-11-05 17:02:38,917    18186    DEBUG    path davs://xfer.cr.cnaf.infn.it:8443/escape/rucio/cnaf_storm/ESCAPE_CERN_TEAM-noise/53/5c/auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K.rucio.upload
2020-11-05 17:02:39,089    WARNING    Upload attempt failed
2020-11-05 17:02:39,089    18186    WARNING    Upload attempt failed
2020-11-05 17:02:39,090    INFO    Exception: An unknown exception occurred.
Details: Checksum mismatch. Source: 09600001 Destination: 9600001
2020-11-05 17:02:39,090    18186    INFO    Exception: An unknown exception occurred.
Details: Checksum mismatch. Source: 09600001 Destination: 9600001
2020-11-05 17:02:39,092    18186    DEBUG    Starting new HTTPS connection (1): escape-rucio.cern.ch:32300
2020-11-05 17:02:39,111    18186    DEBUG    https://escape-rucio.cern.ch:32300 "POST /traces/ HTTP/1.1" 404 205
2020-11-05 17:02:39,113    ERROR    Failed to upload file auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K
2020-11-05 17:02:39,113    18186    ERROR    Failed to upload file auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K
2020-11-05 17:02:39,117    ERROR    None of the given files have been uploaded.
2020-11-05 17:02:39,117    18186    ERROR    None of the given files have been uploaded.
```


Modification
------------
A modification could be:
- if the check fails, verify `adler32` of the file has leading `0`(s);
- if true, remove leading `0`(s) and try again. 

In principle, this will keep things safe as they are, but ensure a re-try for STORM RSEs.

Changed files:
lib/rucio/client/dq2client.py, lib/rucio/client/uploadclient.py, lib/rucio/client/downloadclient.py

diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
index 78b7dda5f..5d1065c7f 100644
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -913,7 +913,7 @@ class DownloadClient:
                     skip_check = item.get('ignore_checksum', False)
                     rucio_checksum = 0 if skip_check else item.get('adler32')
                     local_checksum = 0 if skip_check else adler32(temp_file_path)
-                    if rucio_checksum == local_checksum:
+                    if str(rucio_checksum).lstrip('0') == str(local_checksum).lstrip('0'):
                         item['clientState'] = 'DONE'
                         trace['clientState'] = 'DONE'
                         # remove .part ending
diff --git a/lib/rucio/client/dq2client.py b/lib/rucio/client/dq2client.py
index 5f9aa829a..5065776ed 100644
--- a/lib/rucio/client/dq2client.py
+++ b/lib/rucio/client/dq2client.py
@@ -1585,7 +1585,7 @@ class DQ2Client:
                     guid = '%s-%s-%s-%s-%s' % (guid[0:8], guid[8:12], guid[12:16], guid[16:20], guid[20:32])
                     if guid != did['meta']['guid']:
                         result[lfn] = {'status': False, 'error': FileConsistencyMismatch('guid mismatch DDM %s vs user %s' % (guid, did['meta']['guid']))}
-                    elif meta['adler32'] != did['adler32']:
+                    elif str(meta['adler32']).lstrip('0') != str(did['adler32']).lstrip('0'):
                         result[lfn] = {'status': False, 'error': FileConsistencyMismatch('adler32 mismatch DDM %s vs user %s' % (meta['adler32'], did['adler32']))}
                     elif meta['bytes'] != did['bytes']:
                         result[lfn] = {'status': False, 'error': FileConsistencyMismatch('filesize mismatch DDM %s vs user %s' % (meta['bytes'], did['bytes']))}
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
index 572cfd008..daa64d28d 100644
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -397,7 +397,7 @@ class UploadClient:
             logger(logging.INFO, 'File DID already exists')
             logger(logging.DEBUG, 'local checksum: %s, remote checksum: %s' % (file['adler32'], meta['adler32']))
 
-            if meta['adler32'] != file['adler32']:
+            if str(meta['adler32']).lstrip('0') != str(file['adler32']).lstrip('0'):
                 logger(logging.ERROR, 'Local checksum %s does not match remote checksum %s' % (file['adler32'], meta['adler32']))
 
                 raise DataIdentifierAlreadyExists
@@ -646,7 +646,7 @@ class UploadClient:
                 if rse_settings['verify_checksum'] is not False:
                     if ('adler32' in stats) and ('adler32' in lfn):
                         self.logger(logging.DEBUG, 'Checksum: Expected=%s Found=%s' % (lfn['adler32'], stats['adler32']))
-                        if stats['adler32'] != lfn['adler32']:
+                        if str(stats['adler32']).lstrip('0') != str(lfn['adler32']).lstrip('0'):
                             raise RucioException('Checksum mismatch. Source: %s Destination: %s' % (lfn['adler32'], stats['adler32']))
 
             except Exception as error: