Repository: iterative/dvc

Issue:
state: incompatible across different python versions
### Steps to reproduce

```bash
python3.9 -m dvc diff -v
python3.6 -m dvc diff -v
```

#### Output

```console
$ python3.9 -m dvc diff -v
2021-03-08 16:09:20,120 DEBUG: Check for update is disabled.
2021-03-08 16:09:20,202 DEBUG: state save (24563398, 1611922311010025216, 211598) 90104d9e83cfb825cf45507e90aadd27
2021-03-08 16:09:20,243 DEBUG: state save (24566815, 1611922311020025344, 12238) 94614d6650e062655f9f77507dc9c1f2
2021-03-08 16:09:20,248 DEBUG: state save (24563444, 1611922311016691968, 206122) 9106cda08aa427e73492389a0f17c72d
2021-03-08 16:09:20,417 DEBUG: Analytics is disabled.
$ python3.6 -m dvc diff -v
dvc diff -v
2021-03-08 16:09:42,589 DEBUG: Check for update is disabled.
2021-03-08 16:09:42,714 ERROR: unexpected error - unsupported pickle protocol: 5
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/saugat/repos/iterative/dvc/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/home/saugat/repos/iterative/dvc/dvc/command/diff.py", line 138, in run
    self.args.a_rev, self.args.b_rev, self.args.targets
  File "/home/saugat/repos/iterative/dvc/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/saugat/repos/iterative/dvc/dvc/repo/diff.py", line 45, in diff
    results[rev] = _paths_checksums(self, repo_fs, targets_path_infos)
  File "/home/saugat/repos/iterative/dvc/dvc/repo/diff.py", line 110, in _paths_checksums
    return dict(_output_paths(repo, repo_fs, targets))
  File "/home/saugat/repos/iterative/dvc/dvc/repo/diff.py", line 145, in _output_paths
    yield _to_path(output), _to_checksum(output)
  File "/home/saugat/repos/iterative/dvc/dvc/repo/diff.py", line 133, in _to_checksum
    repo.odb.local, output.path_info, repo.odb.local.fs,
  File "/home/saugat/repos/iterative/dvc/dvc/objects/stage.py", line 152, in stage
    return File.stage(odb, path_info, fs, **kwargs)
  File "/home/saugat/repos/iterative/dvc/dvc/objects/__init__.py", line 68, in stage
    path_info, fs, odb.fs.PARAM_CHECKSUM, odb, **kwargs
  File "/home/saugat/repos/iterative/dvc/dvc/objects/stage.py", line 115, in get_hash
    hash_info = state.get(path_info, fs)
  File "/home/saugat/repos/iterative/dvc/dvc/state.py", line 139, in get
    value = self.md5s.get(inode)
  File "/home/saugat/venvs/dvc/env36/lib/python3.6/site-packages/diskcache/core.py", line 1202, in get
    value = self._disk.fetch(mode, filename, db_value, read)
  File "/home/saugat/venvs/dvc/env36/lib/python3.6/site-packages/diskcache/core.py", line 282, in fetch
    return pickle.load(io.BytesIO(value))
ValueError: unsupported pickle protocol: 5
------------------------------------------------------------
2021-03-08 16:09:44,189 DEBUG: Version info for developers:
DVC version: 2.0.3+70f4c4.mod 
---------------------------------
Platform: Python 3.6.10 on Linux-5.10.11-arch1-1-x86_64-with-arch
Supports: All remotes
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda9
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda9
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-03-08 16:09:44,192 DEBUG: Analytics is disabled.
```

It might be an easy fix as following, but we need to ensure compatibility for the buggy 2.0 release with this new patch as well:
```diff
diff --git a/dvc/state.py b/dvc/state.py
index 3d7b4a30..1b647e21 100644
--- a/dvc/state.py
+++ b/dvc/state.py
@@ -76,10 +76,11 @@ class State(StateBase):  # pylint: disable=too-many-instance-attributes
         if not tmp_dir:
             return
 
-        config = {"eviction_policy": "least-recently-used"}
+        config = {"eviction_policy": "least-recently-used", "disk_pickle_protocol": 4}
         self.links = Cache(directory=os.path.join(tmp_dir, "links"), **config)
         self.md5s = Cache(directory=os.path.join(tmp_dir, "md5s"), **config)
```

Version 5 is only supported on Python3.8 onwards.


Discussion:
My fault, didn't set that option initially :slightly_frowning_face: Let's at least set it properly right now, the worst-case - people will need to `rm -rf .dvc/tmp`, which is not that bad. We could use a backport of pickle5 to ensure the compatibility for existing repos, but that will require monkeypatching, which is not the best too. Might just bite the bullet to at least fix it for the new repos.

Resolving patch:
diff --git a/dvc/state.py b/dvc/state.py
--- a/dvc/state.py
+++ b/dvc/state.py
@@ -76,7 +76,10 @@ def __init__(self, root_dir=None, tmp_dir=None):
         if not tmp_dir:
             return
 
-        config = {"eviction_policy": "least-recently-used"}
+        config = {
+            "eviction_policy": "least-recently-used",
+            "disk_pickle_protocol": 4,
+        }
         self.links = Cache(directory=os.path.join(tmp_dir, "links"), **config)
         self.md5s = Cache(directory=os.path.join(tmp_dir, "md5s"), **config)
 


Test patch: