Repository: iterative/dvc

Issue:
ERROR: unexpected error - list.remove(x): x not in list, if I execute "dvc diff HEAD~",  after  dvc version is updated from 1.1.15 to 2.0.5
# Bug Report

ERROR: unexpected error - list.remove(x): x not in list, if I execute "dvc diff HEAD~", after dvc version is updated from 1.1.15 to 2.0.5

## Description

"dvc diff" command is wrong in version 2.0.5.


### Reproduce

Step list of how to reproduce the bug

1. I made dvc repository in 1.11.15 and push data into GCS.
2. "pip install dvc --upgrade"
3. Create a file in a directory named foo managed by dvc
4. "dvc add foo"
5. "dvc diff HEAD~"
6. shown "ERROR: unexpected error - list.remove(x): x not in list"

### Expected

"dvc diff" command execute as well as 1.11.15.
looks like below.
```
Added:
    bar.txt
```


### Environment information


**Output of `dvc doctor`:**

```console
$ dvc doctor
```

```console
DVC version: 2.0.5 (pip)
---------------------------------
Platform: Python 3.7.9 on Darwin-19.6.0-x86_64-i386-64bit
Supports: gs, hdfs, webhdfs, http, https
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s2
Caches: local
Remotes: gs
Workspace directory: apfs on /dev/disk1s2
Repo: dvc, git
```

**Additional Information (if any):**

part of error message of ```dvc diff --verbose HEAD~``` is below.
```
Traceback (most recent call last):
  File "venv/lib/python3.7/site-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "<reposistory>/venv/lib/python3.7/site-packages/dvc/command/diff.py", line 138, in run
    self.args.a_rev, self.args.b_rev, self.args.targets
  File "<reposistory>/venv/lib/python3.7/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "<reposistory>/venv/lib/python3.7/site-packages/dvc/repo/diff.py", line 79, in diff
    deleted.remove(renamed_item["path"]["old"])
ValueError: list.remove(x): x not in list
```

version 2.0.0 is wrong too.
version 1.11.16 is correct.

Thanks.


Discussion:
Just to confirm, if you use the actual git revision for the previous commit instead of `HEAD~` shorthand, do you still get the same error?
Yes. I executed ``` dvc diff REVISION_1 ``` .
REVISION_1 is git revision like 2ef5b363692e27c786e0fee85dc1b1cb93e32ba5 and it shows the same revision of "HEAD~".

I still got the same error. ```ERROR: unexpected error - list.remove(x): x not in list```

I alse executed ``` dvc diff  REVISION_2  REVISION_1 ``` .
REVISION_1 is the same as HEAD~. 
REVISION_2 is the same as HEAD.

I still got the same error too. ```ERROR: unexpected error - list.remove(x): x not in list```

regards.


Resolving patch:
diff --git a/dvc/repo/diff.py b/dvc/repo/diff.py
--- a/dvc/repo/diff.py
+++ b/dvc/repo/diff.py
@@ -1,5 +1,7 @@
 import logging
 import os
+from collections import defaultdict
+from typing import Dict, List
 
 from dvc.exceptions import PathMissingError
 from dvc.objects.stage import get_file_hash
@@ -190,29 +192,23 @@ def _targets_to_path_infos(repo_fs, targets):
 
 
 def _calculate_renamed(new, old, added, deleted):
-    old_inverted = {}
+    old_inverted: Dict[str, List[str]] = defaultdict(list)
     # It is needed to be dict of lists to cover cases
     # when repo has paths with same hash
     for path, path_hash in old.items():
-        bucket = old_inverted.get(path_hash, None)
-        if bucket is None:
-            old_inverted[path_hash] = [path]
-        else:
-            bucket.append(path)
+        old_inverted[path_hash].append(path)
 
     renamed = []
     for path in added:
         path_hash = new[path]
-        old_paths = old_inverted.get(path_hash, None)
-        if not old_paths:
-            continue
-        old_path = None
-        for tmp_old_path in old_paths:
-            if tmp_old_path in deleted:
-                old_path = tmp_old_path
-                break
-        if not old_path:
+        old_paths = old_inverted[path_hash]
+        try:
+            iterator = enumerate(old_paths)
+            index = next(idx for idx, path in iterator if path in deleted)
+        except StopIteration:
             continue
+
+        old_path = old_paths.pop(index)
         renamed.append(
             {"path": {"old": old_path, "new": path}, "hash": path_hash}
         )


Test patch:
diff --git a/tests/func/test_diff.py b/tests/func/test_diff.py
--- a/tests/func/test_diff.py
+++ b/tests/func/test_diff.py
@@ -549,3 +549,49 @@ def test_diff_rename_file(tmp_dir, scm, dvc, commit_last):
         ],
         "not in cache": [],
     }
+
+
+def test_rename_multiple_files_same_hashes(tmp_dir, scm, dvc):
+    """Test diff by renaming >=2 instances of file with same hashes.
+
+    DVC should be able to detect that they are renames, and should not include
+    them in either of the `added` or the `deleted` section.
+    """
+    tmp_dir.dvc_gen(
+        {"dir": {"foo": "foo", "subdir": {"foo": "foo"}}}, commit="commit #1",
+    )
+    remove(tmp_dir / "dir")
+    # changing foo and subdir/foo to bar and subdir/bar respectively
+    tmp_dir.dvc_gen(
+        {"dir": {"bar": "foo", "subdir": {"bar": "foo"}}}, commit="commit #2"
+    )
+    assert dvc.diff("HEAD~") == {
+        "added": [],
+        "deleted": [],
+        "modified": [
+            {
+                "hash": {
+                    "new": "31b36b3ea5f4485e27f10578c47183b0.dir",
+                    "old": "c7684c8b3b0d28cf80d5305e2d856bfc.dir",
+                },
+                "path": os.path.join("dir", ""),
+            }
+        ],
+        "not in cache": [],
+        "renamed": [
+            {
+                "hash": "acbd18db4cc2f85cedef654fccc4a4d8",
+                "path": {
+                    "new": os.path.join("dir", "bar"),
+                    "old": os.path.join("dir", "foo"),
+                },
+            },
+            {
+                "hash": "acbd18db4cc2f85cedef654fccc4a4d8",
+                "path": {
+                    "new": os.path.join("dir", "subdir", "bar"),
+                    "old": os.path.join("dir", "subdir", "foo"),
+                },
+            },
+        ],
+    }