Repository: iterative/dvc

Issue:
http: use Content-Md5 header if available
We could use it for https://github.com/iterative/dvc/issues/1348 .


Discussion:
@efiop, I'm not sure about this one; check this parragraph from the RFC:

>   The output of the MD5 algorithm is a 128 bit digest.  When viewed in
>   network byte order (big-endian order), this yields a sequence of 16
>   octets of binary data.  These 16 octets are then encoded according to
>   the base64 algorithm in order to obtain the value that is placed in
>   the Content-MD5 field.  Thus, if the application of the MD5 algorithm
>   over the raw data of a MIME entity results in a digest having the
>   (unlikely) value of "Check Integrity!", then that MIME entity's
>   header could contain the field
>
>        Content-MD5:  Q2hlY2sgSW50ZWdyaXR5IQ==

That MD5 doesn't look good... I was expecting something like the result from: `echo -n 'Check Integrity!' | md5sum`.

https://tools.ietf.org/html/rfc1864
For #1348, the stagefile has the MD5 of the downloaded content, so we could use that one to check if it is already cached, alongside with the ETag.
> That MD5 doesn't look good... I was expecting something like the result from: echo -n 'Check Integrity!' | md5sum.

It is just encoded in base64 and md5sum outputs hex. Here is us converting it in GS remove: https://github.com/iterative/dvc/blob/0.22.0/dvc/remote/gs.py#L55


> For #1348, the stagefile has the MD5 of the downloaded content, so we could use that one to check if it is already cached, alongside with the ETag.

Yes, that would be even better, I agree. Especially since our checksum is dos2unix + md5sum, so content-md5 might not always match our checksum.
> > That MD5 doesn't look good... I was expecting something like the result from: echo -n 'Check Integrity!' | md5sum.
> 
> It is just encoded in base64 and md5sum outputs hex. Here is us converting it in GS remove: https://github.com/iterative/dvc/blob/0.22.0/dvc/remote/gs.py#L55

:sweat_smile: Thanks, @efiop.

Just to be clear, if `Content-MD5` header is available, we will store the decoded value as an `ETag` instead of searching for the `ETag` header, right?
Yes, seems right :) This is not top priority though, ETags works perfectly right now.

Resolving patch:
diff --git a/dvc/remote/http.py b/dvc/remote/http.py
--- a/dvc/remote/http.py
+++ b/dvc/remote/http.py
@@ -107,10 +107,14 @@ def _content_length(self, url):
         return self._request('HEAD', url).headers.get('Content-Length')
 
     def _etag(self, url):
-        etag = self._request('HEAD', url).headers.get('ETag')
+        etag = (self._request('HEAD', url).headers.get('ETag')
+                or self._request('HEAD', url).headers.get('Content-MD5'))
 
         if not etag:
-            raise DvcException("could not find an ETag for '{}'".format(url))
+            raise DvcException(
+                "could not find an ETag or Content-MD5 header for '{url}'"
+                .format(url=url)
+            )
 
         if etag.startswith('W/'):
             raise DvcException(


Test patch:
diff --git a/tests/test_repro.py b/tests/test_repro.py
--- a/tests/test_repro.py
+++ b/tests/test_repro.py
@@ -1131,6 +1131,16 @@ def test(self):
         self.assertTrue(os.path.exists(import_output))
         self.assertTrue(filecmp.cmp(import_output, self.FOO, shallow=False))
 
+        self.dvc.remove('imported_file.dvc')
+
+        with StaticFileServer(handler='Content-MD5'):
+            import_url = urljoin(self.remote, self.FOO)
+            import_output = 'imported_file'
+            import_stage = self.dvc.imp(import_url, import_output)
+
+        self.assertTrue(os.path.exists(import_output))
+        self.assertTrue(filecmp.cmp(import_output, self.FOO, shallow=False))
+
         # Run --deps
         with StaticFileServer():
             run_dependency = urljoin(self.remote, self.BAR)
diff --git a/tests/utils/httpd.py b/tests/utils/httpd.py
--- a/tests/utils/httpd.py
+++ b/tests/utils/httpd.py
@@ -17,12 +17,26 @@ def end_headers(self):
         SimpleHTTPRequestHandler.end_headers(self)
 
 
+class ContentMD5Handler(SimpleHTTPRequestHandler):
+    def end_headers(self):
+        file = self.translate_path(self.path)
+
+        if not os.path.isdir(file) and os.path.exists(file):
+            with open(file, 'r') as fd:
+                md5 = hashlib.md5(fd.read().encode('utf8')).hexdigest()
+                self.send_header('Content-MD5', md5)
+
+        SimpleHTTPRequestHandler.end_headers(self)
+
+
 class StaticFileServer:
     __server_lock = threading.Lock()
 
-    def __init__(self):
+    def __init__(self, handler='etag'):
+        handler_class = ETagHandler if handler == 'etag' else ContentMD5Handler
+
         self.__server_lock.acquire()
-        self.httpd = HTTPServer(('localhost', 8000), ETagHandler)
+        self.httpd = HTTPServer(('localhost', 8000), handler_class)
 
     def __enter__(self):
         self.server_thread = threading.Thread(target=self.httpd.serve_forever)