Repository: rucio/rucio

Issue:
STORM-Rucio checksum mismatch when adler32 has leading `0`(s) #3494 #3538
Motivation
----------
During `rucio upload` towards a STORM RSE, a checksum mismatch occurs when `adler32` of the file has leading `0`(s).

```
2020-11-05 17:02:38,917    18186    INFO    Successful upload of temporary file. davs://xfer.cr.cnaf.infn.it:8443/escape/rucio/cnaf_storm/ESCAPE_CERN_TEAM-noise/53/5c/auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K.rucio.upload
2020-11-05 17:02:38,917    18186    DEBUG    path davs://xfer.cr.cnaf.infn.it:8443/escape/rucio/cnaf_storm/ESCAPE_CERN_TEAM-noise/53/5c/auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K.rucio.upload
2020-11-05 17:02:39,089    WARNING    Upload attempt failed
2020-11-05 17:02:39,089    18186    WARNING    Upload attempt failed
2020-11-05 17:02:39,090    INFO    Exception: An unknown exception occurred.
Details: Checksum mismatch. Source: 09600001 Destination: 9600001
2020-11-05 17:02:39,090    18186    INFO    Exception: An unknown exception occurred.
Details: Checksum mismatch. Source: 09600001 Destination: 9600001
2020-11-05 17:02:39,092    18186    DEBUG    Starting new HTTPS connection (1): escape-rucio.cern.ch:32300
2020-11-05 17:02:39,111    18186    DEBUG    https://escape-rucio.cern.ch:32300 "POST /traces/ HTTP/1.1" 404 205
2020-11-05 17:02:39,113    ERROR    Failed to upload file auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K
2020-11-05 17:02:39,113    18186    ERROR    Failed to upload file auto_uploaded_OVDB1ZA7OYTJPw8nWshOloiU3rHgLP0K
2020-11-05 17:02:39,117    ERROR    None of the given files have been uploaded.
2020-11-05 17:02:39,117    18186    ERROR    None of the given files have been uploaded.
```


Modification
------------
A modification could be:
- if the check fails, verify `adler32` of the file has leading `0`(s);
- if true, remove leading `0`(s) and try again. 

In principle, this will keep things safe as they are, but ensure a re-try for STORM RSEs.


Discussion:
Adding @andreaceccanti and @mpatrascoiu
Paul pointed this out: https://www.iana.org/assignments/http-dig-alg/http-dig-alg.xhtml .
Also mentioned in https://tools.ietf.org/html/draft-ietf-httpbis-digest-headers-04#section-13.8 so everybody should normalize (remove/add leading zeroes) while comparing `ADLER32` checksum value (same behaviour has also `CRC32c`). Rucio basically gets exactly the value from `Digest: adler32=xxxxx` header, because Davix doesn't try to do anything clever with checksum output

https://github.com/cern-fts/davix/blob/3c48eabb1a573f90392e1a8c0915fcc2aed90af8/src/utils/checksum_extractor.cpp#L72-L90

On the other hand e.g. for local files `gfal2` tries to be clever and always normalize `ADLER32` output with leading zeroes

https://gitlab.cern.ch/dmc/gfal2/-/blob/develop/src/plugins/file/gfal_file_plugin_main.c#L415

Also for root it seems to me value is always normalized with leading zeroes.
Thanks for the links Petr. I would suggest that we directly normalize the adler32 checksums (Thus, remove the `0`s before comparison). No need to let it fail first, sanitise and then check again.
I suppose this never showed up due to ATLAS not directly using the rucio clients to upload to storm.
@riccardodimaria @mlassnig please coordinate about submitting a PR for this :-) thanks
Hey @bari12 , can I take this up?
I also wanted to ask, do we normalize this directly while computing the adler32 value for the file, or instead normalize only while comparing the values?
Hi @rak108 
Yes sure, please just submit a PR for it and we will have a look. Thanks!

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -913,7 +913,7 @@ def _download_items_aria2c(self, items, aria_rpc, rpc_auth, trace_custom_fields=
                     skip_check = item.get('ignore_checksum', False)
                     rucio_checksum = 0 if skip_check else item.get('adler32')
                     local_checksum = 0 if skip_check else adler32(temp_file_path)
-                    if rucio_checksum == local_checksum:
+                    if str(rucio_checksum).lstrip('0') == str(local_checksum).lstrip('0'):
                         item['clientState'] = 'DONE'
                         trace['clientState'] = 'DONE'
                         # remove .part ending
diff --git a/lib/rucio/client/dq2client.py b/lib/rucio/client/dq2client.py
--- a/lib/rucio/client/dq2client.py
+++ b/lib/rucio/client/dq2client.py
@@ -1585,7 +1585,7 @@ def registerFilesInDataset(self, dsn, lfns=[], guids=[], sizes=[], checksums=[],
                     guid = '%s-%s-%s-%s-%s' % (guid[0:8], guid[8:12], guid[12:16], guid[16:20], guid[20:32])
                     if guid != did['meta']['guid']:
                         result[lfn] = {'status': False, 'error': FileConsistencyMismatch('guid mismatch DDM %s vs user %s' % (guid, did['meta']['guid']))}
-                    elif meta['adler32'] != did['adler32']:
+                    elif str(meta['adler32']).lstrip('0') != str(did['adler32']).lstrip('0'):
                         result[lfn] = {'status': False, 'error': FileConsistencyMismatch('adler32 mismatch DDM %s vs user %s' % (meta['adler32'], did['adler32']))}
                     elif meta['bytes'] != did['bytes']:
                         result[lfn] = {'status': False, 'error': FileConsistencyMismatch('filesize mismatch DDM %s vs user %s' % (meta['bytes'], did['bytes']))}
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -397,7 +397,7 @@ def _register_file(self, file, registered_dataset_dids):
             logger(logging.INFO, 'File DID already exists')
             logger(logging.DEBUG, 'local checksum: %s, remote checksum: %s' % (file['adler32'], meta['adler32']))
 
-            if meta['adler32'] != file['adler32']:
+            if str(meta['adler32']).lstrip('0') != str(file['adler32']).lstrip('0'):
                 logger(logging.ERROR, 'Local checksum %s does not match remote checksum %s' % (file['adler32'], meta['adler32']))
 
                 raise DataIdentifierAlreadyExists
@@ -646,7 +646,7 @@ def _upload_item(self, rse_settings, rse_attributes, lfn, source_dir=None, force
                 if rse_settings['verify_checksum'] is not False:
                     if ('adler32' in stats) and ('adler32' in lfn):
                         self.logger(logging.DEBUG, 'Checksum: Expected=%s Found=%s' % (lfn['adler32'], stats['adler32']))
-                        if stats['adler32'] != lfn['adler32']:
+                        if str(stats['adler32']).lstrip('0') != str(lfn['adler32']).lstrip('0'):
                             raise RucioException('Checksum mismatch. Source: %s Destination: %s' % (lfn['adler32'], stats['adler32']))
 
             except Exception as error:


Test patch: