Repository: iterative/dvc

Issue:
straight-to-remote: using different block size to compute md5 than the local hashing
We use a complete chunk to determine whether the file is text or not for the straight-to-remote hashing.
https://github.com/iterative/dvc/blob/b9df4451b196082d66e15a22ecc7ca5660e02d47/dvc/utils/stream.py#L91

Whereas, we use the first 512 bytes for the heuristics.
https://github.com/iterative/dvc/blob/b9df4451b196082d66e15a22ecc7ca5660e02d47/dvc/istextfile.py#L25

This might cause issues where the file hashes do not match when the file is checked out later.
```
2021-06-14 15:48:03,158 WARNING: corrupted cache file 'http://localhost:8080/remote.php/dav/files/admin/dvc-cache/27/942e2b84fe17ce3778fd795b1cedc1'.
```

Note that I found this while trying to add a PDF file straight to the cache. The checks for PDF are always going to be tricky,
so I am not expecting that to be fixed here, but we should definitely use the same amount of chunk to check for binary or texts. 


Comments:
This would have been caught if we were running all of our tests with remote using `verify = True`. 🙁 
The transfer logic that to-remote/to-cache uses depends on the `fs.CHUNK_SIZE`, which seems to be always greater than the `512`. Unless I am missing something, we could potentially just pass `chunk[:512]`
@isidentical, it seems to use `read1` if it exists, so maybe read the first 512 bytes before hashing?

https://github.com/iterative/dvc/blob/b9df4451b196082d66e15a22ecc7ca5660e02d47/dvc/utils/stream.py#L84-L85
> so maybe read the first 512 bytes before hashing?

So you mean we should pre-emptively read the 512 bytes, and cache it in the initialization? That seems to be rather complicated:
```py
        chunk = b""
        if self.tell() <= 512:
            if n == -1:
                chunk += self._initial_chunk[self.tell():]
            else:
                chunk += self._initial_chunk[self.tell():self.tell()+n]
                n -= len(chunk)
        chunk += self._reader(n)
```


Patch:
diff --git a/dvc/istextfile.py b/dvc/istextfile.py
--- a/dvc/istextfile.py
+++ b/dvc/istextfile.py
@@ -5,6 +5,7 @@
 
 
 TEXT_CHARS = bytes(range(32, 127)) + b"\n\r\t\f\b"
+DEFAULT_CHUNK_SIZE = 512
 
 
 def istextblock(block):
@@ -22,7 +23,7 @@ def istextblock(block):
     return float(len(nontext)) / len(block) <= 0.30
 
 
-def istextfile(fname, fs, blocksize=512):
+def istextfile(fname, fs, blocksize=DEFAULT_CHUNK_SIZE):
     """Uses heuristics to guess whether the given file is text or binary,
     by reading a single block of bytes from the file.
     If more than 30% of the chars in the block are non-text, or there
diff --git a/dvc/utils/stream.py b/dvc/utils/stream.py
--- a/dvc/utils/stream.py
+++ b/dvc/utils/stream.py
@@ -4,7 +4,7 @@
 from funcy import cached_property
 
 from dvc.hash_info import HashInfo
-from dvc.istextfile import istextblock
+from dvc.istextfile import DEFAULT_CHUNK_SIZE, istextblock
 from dvc.utils import dos2unix
 
 
@@ -89,7 +89,7 @@ def _reader(self):
     def read(self, n=-1):
         chunk = self._reader(n)
         if self.is_text_file is None:
-            self.is_text_file = istextblock(chunk)
+            self.is_text_file = istextblock(chunk[:DEFAULT_CHUNK_SIZE])
 
         if self.is_text_file:
             data = dos2unix(chunk)


Test patch:
diff --git a/tests/unit/utils/test_stream.py b/tests/unit/utils/test_stream.py
--- a/tests/unit/utils/test_stream.py
+++ b/tests/unit/utils/test_stream.py
@@ -1,4 +1,7 @@
+import pytest
+
 from dvc.fs.local import LocalFileSystem
+from dvc.istextfile import DEFAULT_CHUNK_SIZE, istextfile
 from dvc.utils import file_md5
 from dvc.utils.stream import HashedStreamReader
 
@@ -46,3 +49,26 @@ def test_hashed_stream_reader_as_chunks(tmp_dir):
     hex_digest = file_md5(foo, LocalFileSystem())
     assert not stream_reader.is_text_file
     assert hex_digest == stream_reader.hash_info.value
+
+
+@pytest.mark.parametrize(
+    "contents",
+    [b"x" * DEFAULT_CHUNK_SIZE + b"\x00", b"clean", b"not clean \x00"],
+)
+def test_hashed_stream_reader_compatibility(tmp_dir, contents):
+    # Always read more than the DEFAULT_CHUNK_SIZE (512 bytes).
+    # This imitates the read actions performed by upload_fobj.
+    chunk_size = DEFAULT_CHUNK_SIZE * 2
+
+    tmp_dir.gen("data", contents)
+    data = tmp_dir / "data"
+
+    with open(data, "rb") as fobj:
+        stream_reader = HashedStreamReader(fobj)
+        stream_reader.read(chunk_size)
+
+    local_fs = LocalFileSystem()
+    hex_digest = file_md5(data, local_fs)
+
+    assert stream_reader.is_text_file is istextfile(data, local_fs)
+    assert stream_reader.hash_info.value == hex_digest