Repository: iterative/dvc

Issue:
push: Unsupported pickle protocol: 5 on python 3.7.11
# Bug Report

## Description

When attempting to push to remote (Google Cloud Storage) using DVC version `2.10.2` and Python version `3.7.11` I encounter the following error:

```console
$ dvc push     
ERROR: unexpected error - unsupported pickle protocol: 5              

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
```

In my tests on python 3.8 this issue does not happen.

### Reproduce

Install DVC version `2.10.2` with Python version `3.7.11` and attempt to push to remote.

### Expected

Expected to pickle successfully and push the pickled data to remote (Google Cloud Storage).

### Environment information

```console
$ dvc doctor
DVC version: 2.10.2 (conda)
---------------------------------
Platform: Python 3.7.11 on Darwin-21.2.0-x86_64-i386-64bit
Supports:
        gs (gcsfs = 2022.1.0),
        webhdfs (fsspec = 2022.1.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: gs
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git
```



Comments:
@jnissin, can you share the output from `dvc push -v`? 
Here you go @skshetry :

```console
$ dvc push -v
2022-05-11 13:48:51,201 DEBUG: Preparing to transfer data from '/Users/***/Documents/code/work/***/.dvc/cache' to 'gs://valimo-dvc-storage'
2022-05-11 13:48:51,201 DEBUG: Preparing to collect status from 'gs://valimo-dvc-storage'
2022-05-11 13:48:51,201 DEBUG: Collecting status from 'gs://valimo-dvc-storage'
2022-05-11 13:48:51,204 ERROR: unexpected error - unsupported pickle protocol: 5
------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/cli/__init__.py", line 90, in main
    ret = cmd.do_run()
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/commands/data_sync.py", line 68, in run
    glob=self.args.glob,
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/repo/push.py", line 69, in push
    obj_ids, jobs, remote=remote, odb=odb or dest_odb
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data_cloud.py", line 91, in push
    cache_odb=self.repo.odb.local,
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data/transfer.py", line 155, in transfer
    src, dest, obj_ids, check_deleted=False, jobs=jobs, **kwargs
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data/status.py", line 181, in compare_status
    dest, obj_ids, index=dest_index, jobs=jobs, **kwargs
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data/status.py", line 138, in status
    odb, index, dir_objs, name, cache_odb, jobs=jobs
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data/status.py", line 36, in _indexed_dir_hashes
    indexed_dirs = set(index.dir_hashes())
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data/db/index.py", line 119, in dir_hashes
    yield from (hash_ for hash_, is_dir in self.index.items() if is_dir)
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/dvc/data/db/index.py", line 119, in <genexpr>
    yield from (hash_ for hash_, is_dir in self.index.items() if is_dir)
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/_collections_abc.py", line 744, in __iter__
    yield (key, self._mapping[key])
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/diskcache/persistent.py", line 730, in __getitem__
    return self._cache[key]
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/diskcache/core.py", line 1232, in __getitem__
    value = self.get(key, default=ENOVAL, retry=True)
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/diskcache/core.py", line 1173, in get
    value = self._disk.fetch(mode, filename, db_value, read)
  File "/opt/anaconda3/envs/valimo-local/lib/python3.7/site-packages/diskcache/core.py", line 285, in fetch
    return pickle.load(io.BytesIO(value))
ValueError: unsupported pickle protocol: 5
------------------------------------------------------------
2022-05-11 13:48:51,316 DEBUG: Removing '/Users/***/Documents/code/work/***/.mJCQTQ7VWcdKbQvxRdcp2y.tmp'
2022-05-11 13:48:51,318 DEBUG: Removing '/Users/***/Documents/code/work/***/.mJCQTQ7VWcdKbQvxRdcp2y.tmp'
2022-05-11 13:48:51,318 DEBUG: Removing '/Users/***/Documents/code/work/***/.mJCQTQ7VWcdKbQvxRdcp2y.tmp'
2022-05-11 13:48:51,318 DEBUG: Removing '/Users/***/Documents/code/work/***/.dvc/cache/.68G3bcZEzc4zjdHpDsNqYj.tmp'
2022-05-11 13:48:51,322 DEBUG: Version info for developers:
DVC version: 2.10.2 (conda)
---------------------------------
Platform: Python 3.7.11 on Darwin-21.2.0-x86_64-i386-64bit
Supports:
        gs (gcsfs = 2022.1.0),
        webhdfs (fsspec = 2022.1.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: gs
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-05-11 13:48:51,326 DEBUG: Analytics is enabled.
2022-05-11 13:48:51,457 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/4p/0n_mm8js23x1vhydv_3yq8b40000gq/T/tmp_4ld4dv3']'
2022-05-11 13:48:51,459 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/4p/0n_mm8js23x1vhydv_3yq8b40000gq/T/tmp_4ld4dv3']'
```
This seems like https://dvc.org/doc/user-guide/troubleshooting#pickle but we are not catching the exception in `pull` in order to redirect users to the link:

https://github.com/iterative/dvc/blob/47d10c4a8b9bdc18dec17a9380f0593e43383c4c/dvc/utils/decorators.py#L13-L30
True, I can confirm that the following fixed the issue:

```
rm -r .dvc/tmp/index .dvc/tmp/md5s .dvc/tmp/links
```

This is probably due to the fact that we have python 3.8 running in our pre-commit hooks environment.
@jnissin this behavior (but not the incorrect error message) is expected if you are mixing python3.8 and 3.7 usage within the same repo. See the docs linked by @daavoo, you will need to remove `.dvc/tmp/index` before running in python 3.7 in this case.
Alright, I'll try to switch our pre-commit environment to use python 3.7 as well. Thanks a lot!
@jnissin Btw, did you migrate from an older dvc version recently? We had a bug where we didn't set pickle version https://github.com/iterative/dvc/pull/7222 , but it was fixed 5 months back in 2.9.4

Patch:
diff --git a/dvc/cli/__init__.py b/dvc/cli/__init__.py
--- a/dvc/cli/__init__.py
+++ b/dvc/cli/__init__.py
@@ -36,6 +36,43 @@ def parse_args(argv=None):
     return args
 
 
+def _log_exceptions(exc: Exception):
+    """Try to log some known exceptions, that are not DVCExceptions."""
+    from dvc.objects.cache import DiskError
+    from dvc.utils import error_link
+
+    if isinstance(exc, DiskError):
+        from dvc.utils import relpath
+
+        directory = relpath(exc.directory)
+        logger.exception(
+            f"Could not open pickled '{exc.type}' cache.\n"
+            f"Remove the '{directory}' directory and then retry this command."
+            f"\nSee {error_link('pickle')} for more information.",
+            extra={"tb_only": True},
+        )
+        return
+
+    if isinstance(exc, OSError):
+        import errno
+
+        if exc.errno == errno.EMFILE:
+            logger.exception(
+                "too many open files, please visit "
+                "{} to see how to handle this "
+                "problem".format(error_link("many-files")),
+                extra={"tb_only": True},
+            )
+            return
+
+    from dvc.info import get_dvc_info
+    from dvc.logger import FOOTER
+
+    logger.exception("unexpected error")
+    logger.debug("Version info for developers:\n%s", get_dvc_info())
+    logger.info(FOOTER)
+
+
 def main(argv=None):  # noqa: C901
     """Main entry point for dvc CLI.
 
@@ -48,7 +85,7 @@ def main(argv=None):  # noqa: C901
     from dvc._debug import debugtools
     from dvc.config import ConfigError
     from dvc.exceptions import DvcException, NotDvcRepoError
-    from dvc.logger import FOOTER, disable_other_loggers
+    from dvc.logger import disable_other_loggers
 
     # NOTE: stderr/stdout may be closed if we are running from dvc.daemon.
     # On Linux we directly call cli.main after double forking and closing
@@ -104,26 +141,7 @@ def main(argv=None):  # noqa: C901
         ret = 254
     except Exception as exc:  # noqa, pylint: disable=broad-except
         # pylint: disable=no-member
-        import errno
-
-        if isinstance(exc, OSError) and exc.errno == errno.EMFILE:
-            from dvc.utils import error_link
-
-            logger.exception(
-                "too many open files, please visit "
-                "{} to see how to handle this "
-                "problem".format(error_link("many-files")),
-                extra={"tb_only": True},
-            )
-        else:
-            from dvc.info import get_dvc_info
-
-            logger.exception("unexpected error")
-
-            dvc_info = get_dvc_info()
-            logger.debug("Version info for developers:\n%s", dvc_info)
-
-            logger.info(FOOTER)
+        _log_exceptions(exc)
         ret = 255
 
     try:
diff --git a/dvc/data/db/index.py b/dvc/data/db/index.py
--- a/dvc/data/db/index.py
+++ b/dvc/data/db/index.py
@@ -4,7 +4,6 @@
 from typing import TYPE_CHECKING, Iterable, Set
 
 from dvc.objects.errors import ObjectDBError
-from dvc.utils.decorators import with_diskcache
 
 if TYPE_CHECKING:
     from dvc.types import StrPath
@@ -89,49 +88,40 @@ def __init__(
         tmp_dir: "StrPath",
         name: str,
     ):  # pylint: disable=super-init-not-called
-        from diskcache import Cache, Index
-
         from dvc.fs.local import LocalFileSystem
+        from dvc.objects.cache import Cache, Index
         from dvc.utils.fs import makedirs
 
         self.index_dir = os.path.join(tmp_dir, self.INDEX_DIR, name)
         makedirs(self.index_dir, exist_ok=True)
         self.fs = LocalFileSystem()
-        self.index = Index.fromcache(
-            Cache(
-                self.index_dir,
-                disk_pickle_protocol=4,
-                eviction_policy="none",
-            )
+        cache = Cache(
+            self.index_dir, eviction_policy="none", disk_type="index"
         )
+        self.index = Index.fromcache(cache)
 
-    @with_diskcache(name="index")
     def __iter__(self):
         return iter(self.index)
 
-    @with_diskcache(name="index")
     def __contains__(self, hash_):
         return hash_ in self.index
 
-    @with_diskcache(name="index")
     def dir_hashes(self):
         """Iterate over .dir hashes stored in the index."""
         yield from (hash_ for hash_, is_dir in self.index.items() if is_dir)
 
-    @with_diskcache(name="index")
     def clear(self):
         """Clear this index (to force re-indexing later)."""
-        from diskcache import Timeout
+        from dvc.objects.cache import Timeout
 
         try:
             self.index.clear()
         except Timeout as exc:
             raise ObjectDBError("Failed to clear ODB index") from exc
 
-    @with_diskcache(name="index")
     def update(self, dir_hashes: Iterable[str], file_hashes: Iterable[str]):
         """Update this index, adding the specified hashes."""
-        from diskcache import Timeout
+        from dvc.objects.cache import Timeout
 
         try:
             with self.index.transact():
@@ -143,7 +133,6 @@ def update(self, dir_hashes: Iterable[str], file_hashes: Iterable[str]):
         except Timeout as exc:
             raise ObjectDBError("Failed to update ODB index") from exc
 
-    @with_diskcache(name="index")
     def intersection(self, hashes: Set[str]):
         """Iterate over values from `hashes` which exist in the index."""
         yield from hashes.intersection(self.index.keys())
diff --git a/dvc/objects/cache.py b/dvc/objects/cache.py
new file mode 100644
--- /dev/null
+++ b/dvc/objects/cache.py
@@ -0,0 +1,61 @@
+import os
+import pickle
+from functools import wraps
+from typing import Any
+
+import diskcache
+from diskcache import Disk as disk
+from diskcache import Index  # noqa: F401, pylint: disable=unused-import
+from diskcache import Timeout  # noqa: F401, pylint: disable=unused-import
+
+# pylint: disable=redefined-builtin
+
+
+class DiskError(Exception):
+    def __init__(self, directory: str, type: str) -> None:
+        self.directory = directory
+        self.type = type
+        super().__init__(f"Could not open disk '{type}' in {directory}")
+
+
+def translate_pickle_error(fn):
+    @wraps(fn)
+    def wrapped(self, *args, **kwargs):
+        try:
+            return fn(self, *args, **kwargs)
+        except (pickle.PickleError, ValueError) as e:
+            if isinstance(e, ValueError) and "pickle protocol" not in str(e):
+                raise
+            # pylint: disable=protected-access
+            raise DiskError(self._directory, type=self._type) from e
+
+    return wrapped
+
+
+class Disk(disk):
+    """Reraise pickle-related errors as DiskError."""
+
+    def __init__(self, *args, type=None, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._type = type or os.path.basename(self._directory)
+
+    put = translate_pickle_error(disk.put)
+    get = translate_pickle_error(disk.get)
+    store = translate_pickle_error(disk.store)
+    fetch = translate_pickle_error(disk.fetch)
+
+
+class Cache(diskcache.Cache):
+    """Extended to handle pickle errors and use a constant pickle protocol."""
+
+    def __init__(
+        self,
+        directory: str = None,
+        timeout: int = 60,
+        disk: disk = Disk,  # pylint: disable=redefined-outer-name
+        **settings: Any,
+    ) -> None:
+        settings.setdefault("disk_pickle_protocol", 4)
+        super().__init__(
+            directory=directory, timeout=timeout, disk=Disk, **settings
+        )
diff --git a/dvc/objects/state.py b/dvc/objects/state.py
--- a/dvc/objects/state.py
+++ b/dvc/objects/state.py
@@ -7,7 +7,6 @@
 from dvc.fs.local import LocalFileSystem
 from dvc.fs.system import inode as get_inode
 from dvc.utils import relpath
-from dvc.utils.decorators import with_diskcache
 
 from .hash_info import HashInfo
 from .utils import get_mtime_and_size
@@ -49,7 +48,7 @@ def save_link(self, path, fs):
 
 class State(StateBase):  # pylint: disable=too-many-instance-attributes
     def __init__(self, root_dir=None, tmp_dir=None, dvcignore=None):
-        from diskcache import Cache
+        from .cache import Cache
 
         super().__init__()
 
@@ -60,10 +59,7 @@ def __init__(self, root_dir=None, tmp_dir=None, dvcignore=None):
         if not tmp_dir:
             return
 
-        config = {
-            "eviction_policy": "least-recently-used",
-            "disk_pickle_protocol": 4,
-        }
+        config = {"eviction_policy": "least-recently-used"}
         self.links = Cache(directory=os.path.join(tmp_dir, "links"), **config)
         self.md5s = Cache(directory=os.path.join(tmp_dir, "md5s"), **config)
 
@@ -71,7 +67,6 @@ def close(self):
         self.md5s.close()
         self.links.close()
 
-    @with_diskcache(name="md5s")
     def save(self, path, fs, hash_info):
         """Save hash for the specified path info.
 
@@ -96,7 +91,6 @@ def save(self, path, fs, hash_info):
 
         self.md5s[inode] = (mtime, str(size), hash_info.value)
 
-    @with_diskcache(name="md5s")
     def get(self, path, fs):
         """Gets the hash for the specified path info. Hash will be
         retrieved from the state database if available.
@@ -127,7 +121,6 @@ def get(self, path, fs):
 
         return Meta(size=size), HashInfo("md5", value[2])
 
-    @with_diskcache(name="links")
     def save_link(self, path, fs):
         """Adds the specified path to the list of links created by dvc. This
         list is later used on `dvc checkout` to cleanup old links.
@@ -149,7 +142,6 @@ def save_link(self, path, fs):
         with self.links as ref:
             ref[relative_path] = (inode, mtime)
 
-    @with_diskcache(name="links")
     def get_unused_links(self, used, fs):
         """Removes all saved links except the ones that are used.
 
@@ -177,7 +169,6 @@ def get_unused_links(self, used, fs):
 
         return unused
 
-    @with_diskcache(name="links")
     def remove_links(self, unused, fs):
         if not isinstance(fs, LocalFileSystem):
             return
diff --git a/dvc/utils/decorators.py b/dvc/utils/decorators.py
deleted file mode 100644
--- a/dvc/utils/decorators.py
+++ /dev/null
@@ -1,30 +0,0 @@
-import pickle
-from typing import Callable, TypeVar
-
-from funcy import decorator
-
-from dvc.exceptions import DvcException
-
-from . import format_link
-
-_R = TypeVar("_R")
-
-
-@decorator
-def with_diskcache(call: Callable[..., _R], name: str) -> _R:
-    try:
-        return call()
-    except (pickle.PickleError, ValueError) as exc:
-        if isinstance(exc, ValueError) and not str(exc).startswith(
-            "pickle protocol"
-        ):
-            raise
-        link = format_link(
-            "https://dvc.org/doc/user-guide/troubleshooting#pickle"
-        )
-        msg = (
-            f"Could not open pickled '{name}' cache. Remove the "
-            f"'.dvc/tmp/{name}' directory and then retry this command. "
-            f"See {link} for more information."
-        )
-        raise DvcException(msg) from exc


Test patch:
diff --git a/tests/unit/cli/__init__.py b/tests/unit/cli/__init__.py
new file mode 100644
diff --git a/tests/unit/cli/test_main.py b/tests/unit/cli/test_main.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/cli/test_main.py
@@ -0,0 +1,26 @@
+from argparse import Namespace
+
+from funcy import raiser
+
+from dvc.cli import main
+from dvc.objects.cache import DiskError
+
+
+def test_state_pickle_errors_are_correctly_raised(tmp_dir, caplog, mocker):
+    path = tmp_dir / "dir" / "test"
+    mocker.patch(
+        "dvc.cli.parse_args",
+        return_value=Namespace(
+            func=raiser(DiskError(path, "md5s")),
+            quiet=False,
+            verbose=True,
+        ),
+    )
+
+    assert main() == 255
+    assert (
+        f"Could not open pickled 'md5s' cache.\n"
+        f"Remove the '{path.relative_to(tmp_dir)}' directory "
+        "and then retry this command.\n"
+        "See <https://error.dvc.org/pickle> for more information."
+    ) in caplog.text
diff --git a/tests/unit/utils/test_decorators.py b/tests/unit/objects/test_cache.py
similarity index 52%
rename from tests/unit/utils/test_decorators.py
rename to tests/unit/objects/test_cache.py
--- a/tests/unit/utils/test_decorators.py
+++ b/tests/unit/objects/test_cache.py
@@ -1,27 +1,29 @@
 import pickle
 from typing import Any
 
-import diskcache
 import pytest
 
-from dvc.exceptions import DvcException
-from dvc.utils.decorators import with_diskcache
+from dvc.objects.cache import Cache, DiskError
 
 
-@with_diskcache(name="test")
-def set_value(cache: diskcache.Cache, key: str, value: Any) -> Any:
+def set_value(cache: Cache, key: str, value: Any) -> Any:
     cache[key] = value
     return cache[key]
 
 
-def test_pickle_protocol_error(tmp_dir):
-    with pytest.raises(DvcException) as exc:
-        with diskcache.Cache(
-            directory=(tmp_dir / "test"),
-            disk_pickle_protocol=pickle.HIGHEST_PROTOCOL + 1,
-        ) as cache:
-            set_value(cache, "key", ("value1", "value2"))
-        assert "troubleshooting#pickle" in str(exc)
+@pytest.mark.parametrize("disk_type", [None, "test"])
+def test_pickle_protocol_error(tmp_dir, disk_type):
+    directory = tmp_dir / "test"
+    cache = Cache(
+        directory,
+        disk_pickle_protocol=pickle.HIGHEST_PROTOCOL + 1,
+        disk_type=disk_type,
+    )
+    with pytest.raises(DiskError) as exc, cache as cache:
+        set_value(cache, "key", ("value1", "value2"))
+    assert exc.value.directory == str(directory)
+    assert exc.value.type == "test"
+    assert f"Could not open disk 'test' in {directory}" == str(exc.value)
 
 
 @pytest.mark.parametrize(
@@ -32,12 +34,12 @@ def test_pickle_protocol_error(tmp_dir):
     ],
 )
 def test_pickle_backwards_compat(tmp_dir, proto_a, proto_b):
-    with diskcache.Cache(
+    with Cache(
         directory=(tmp_dir / "test"),
         disk_pickle_protocol=proto_a,
     ) as cache:
         set_value(cache, "key", ("value1", "value2"))
-    with diskcache.Cache(
+    with Cache(
         directory=(tmp_dir / "test"),
         disk_pickle_protocol=proto_b,
     ) as cache: