Repository: iterative/dvc

Issue:
`exp show`: lockless issues
# Bug Report

## Description

After the recent changes to make `exp show` lockless I am seeing intermittent issues with the data returned during experiment runs.

The issues that I have seen so far are as follows:

### Running checkpoint experiments from the queue

1. `exp show` fails when running experiments from the queue with `dulwich.porcelain.DivergedBranches` errors.
2. @pmrowla tried to patch the above but the change led to [this behaviour](https://github.com/iterative/dvc/pull/7684#issuecomment-1116867478).

Please take a look at the above behaviour and LMK what you think. I do not anticipate there being an easy fix. IMO we should consider "dropping support" for live tracking experiments run from the queue until the `DVC` mechanics have been updated.

### Running checkpoint experiment in the workspace

1. `exp show` returns a single dict for a set of checkpoints during an experiment. This happens intermittently and breaks our "live experiment" tracking.
2. `exp show` shows running experiments as not running mid-run.



### Reproduce

<!--
Step list of how to reproduce the bug
-->

Run a checkpoint experiment and monitor the output of `exp show`.

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.9.9 on macOS-12.3.1-x86_64-i386-64bit
Supports:
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc (subdir), git
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->

I'll continue to add issues here as I find them. I have discussed with @pmrowla already. Wanted to raise the visibility by raising an issue to discuss possible mitigation and the priority for fixes.

Thanks



Comments:
I managed to catch the behaviour of a checkpoint experiment being collapsed to a single record without a name (and then returning to all checkpoint):

https://user-images.githubusercontent.com/37993418/166836504-6e839a0f-6d8f-4226-835c-6f0203b47412.mov


For the problem in `Running checkpoint experiments from the queue`

https://github.com/iterative/dvc/pull/7684#issuecomment-1116867478

Unexpected behaviour:

> 1. Only a couple of checkpoints are shown for each experiment until they are both finished and all are then copied to the workspace.
> 2. The running experiment record is shown in a different record to the rest of the experiment.
> 3. The running experiment record returns to queued.


Problems 2` and `3` had already been solved in #8369 

For the problems in `Running checkpoint experiment in the workspace`
> 1. exp show returns a single dict for a set of checkpoints during an experiment. This happens intermittently and breaks our "live experiment" tracking.
> 2. exp show shows running experiments as not running mid-run.


Problem 2` also had already been solved in #8369 and for problem 1, I tried and it still exists. 


So we can focus on problem 1 in both sections. BTW, the epochs finished so quickly in the demo (less than 1S) and that makes the live tracking can not be shown one by one.

For the problem.

1. Only a couple of checkpoints are shown for each experiment until they are both finished and all are then copied to the workspace.

I do some investigation on this problem yesterday. The problem comes from the refs downloaded from

https://github.com/iterative/scmrepo/blob/d175b923c76494c9023ee3581b349191ea2c8a6f/src/scmrepo/git/backend/dulwich/__init__.py#L658-L668 

```python
        try:
            fetch_result = client.fetch(
                path,
                self.repo,
                progress=DulwichProgressReporter(progress)
                if progress
                else None,
                determine_wants=determine_wants,
            )
        except NotGitRepository as exc:
            raise SCMError(f"Git failed to fetch ref from '{url}'") from exc

```

will cause 

https://github.com/iterative/scmrepo/blob/d175b923c76494c9023ee3581b349191ea2c8a6f/src/scmrepo/git/backend/dulwich/__init__.py#L671-L681

```python
        for (lh, rh, _) in fetch_refs:
            refname = os.fsdecode(rh)
            if rh in self.repo.refs:
                if self.repo.refs[rh] == fetch_result.refs[lh]:
                    result[refname] = SyncStatus.UP_TO_DATE
                    continue
                try:
                    check_diverged(
                        self.repo, self.repo.refs[rh], fetch_result.refs[lh]
                    )
```

raise `DivergedBranches`, and the problem behind this error is that the sha value `fetch_result.refs[lh]` does not exist in the repo and will cause `KeyError`. This will cause any updates (fetching) fails, only after the training progress is finished can the fetching succeed. This might come from the new commits coming to fast in the temp workspace because if we add a time gap into the training progress, for example, adding `time.sleep(5)` in each training epoch, the fetching can success and the training can be shown in living progress.

<img width="1674" alt="image" src="https://user-images.githubusercontent.com/6745454/199873215-edb73ab5-8d65-4871-a25f-f6a7fbc42b9a.png">

So if we want to solve this problem completely we might need to go deep into the `dulwich`'s fetching progress and make sure the revision returned is also properly downloaded and exists. And if I use force update here, 

https://github.com/iterative/dvc/blob/43a8eab2e053b072b93d7b399ce0678cb00f138e/dvc/repo/experiments/queue/utils.py#L41-L42

It will raise `Invalid commit` Exception.
might be solved after #8477 if we use `pygit` as new backends for fetch.
I did some research today, and found that the problem I force it to update comes from we are using different backends, the `scm.resolve_commit` is `pygit` backend and it can not properly resolve the force updated commit fetched by from `dulwich` . 

Confirmed that the problem comes from Dulwich. because I saw the ref fetched by Dulwich invalid.

```bash
git commit                                                                                                                                           
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: refs/exps/2a/1c3a54209af04dda29df9fc9309d8e93fcaf45/exp-710c6 does not point to a valid object!
```

Patch:
diff --git a/dvc/repo/experiments/queue/tempdir.py b/dvc/repo/experiments/queue/tempdir.py
--- a/dvc/repo/experiments/queue/tempdir.py
+++ b/dvc/repo/experiments/queue/tempdir.py
@@ -77,9 +77,8 @@ def iter_active(self) -> Generator[QueueEntry, None, None]:
         for stash_rev in self.proc:
             infofile = self.get_infofile_path(stash_rev)
             executor_info = ExecutorInfo.load_json(infofile)
-            if (
-                not executor_info.status <= TaskStatus.SUCCESS
-                and os.path.exists(executor_info.root_dir)
+            if executor_info.status <= TaskStatus.SUCCESS and os.path.exists(
+                executor_info.root_dir
             ):
                 yield QueueEntry(
                     self.repo.root_dir,
diff --git a/dvc/repo/experiments/queue/utils.py b/dvc/repo/experiments/queue/utils.py
--- a/dvc/repo/experiments/queue/utils.py
+++ b/dvc/repo/experiments/queue/utils.py
@@ -41,7 +41,7 @@ def fetch_running_exp_from_temp_dir(
         if info.git_url and fetch_refs and info.status > TaskStatus.PREPARING:
 
             def on_diverged(_ref: str, _checkpoint: bool):
-                return False
+                return True
 
             executor = TempDirExecutor.from_info(info)
             try:
diff --git a/dvc/repo/experiments/show.py b/dvc/repo/experiments/show.py
--- a/dvc/repo/experiments/show.py
+++ b/dvc/repo/experiments/show.py
@@ -8,8 +8,10 @@
     Any,
     Callable,
     Dict,
+    Iterable,
     List,
     Optional,
+    Set,
     Tuple,
     Union,
 )
@@ -24,6 +26,7 @@
 from .refs import ExpRefInfo
 
 if TYPE_CHECKING:
+    from scmrepo.git import Git
     from scmrepo.git.objects import GitCommit
 
     from dvc.repo import Repo
@@ -134,6 +137,7 @@ def _collect_complete_experiment(
 
     checkpoint: bool = True if len(revs) > 1 else False
     prev = ""
+
     for rev in revs:
         status = ExpStatus.Running if rev in running else ExpStatus.Success
         collected_exp = collect_experiment_commit(
@@ -215,41 +219,55 @@ def _collect_branch(
     return results
 
 
-def get_names(repo: "Repo", result: Dict[str, Dict[str, Any]]):
+def get_branch_names(
+    scm: "Git", baseline_set: Iterable[str]
+) -> Dict[str, Optional[str]]:
+    names: Dict[str, Optional[str]] = {}
+    for base in [
+        f"refs/exps/{baseline[:2]}/{baseline[2:]}/"
+        for baseline in baseline_set
+    ] + ["refs/heads/", "refs/tags/"]:
+        for ref in scm.iter_refs(base=base):
+            if ref:
+                try:
+                    rev = scm.get_ref(ref)
+                    names[rev] = ref[len(base) :]
+                except KeyError:
+                    logger.debug("unreosolved ref %s", ref)
+    logger.debug("found refs for revs: %s", names)
+    return names
+
+
+def update_names(
+    repo: "Repo",
+    branch_names: Dict[str, Optional[str]],
+    result: Dict[str, Dict[str, Any]],
+):
 
     rev_set = set()
-    baseline_set = set()
     for baseline in result:
         for rev in result[baseline]:
             if rev == "baseline":
                 rev = baseline
-                baseline_set.add(rev)
             if rev != "workspace":
                 rev_set.add(rev)
 
-    names: Dict[str, Optional[str]] = {}
-    for base in ("refs/tags/", "refs/heads/"):
-        if rev_set:
-            names.update(
-                (rev, ref[len(base) :])
-                for rev, ref in repo.scm.describe(
-                    baseline_set, base=base
-                ).items()
-                if ref is not None
-            )
-            rev_set.difference_update(names.keys())
+    if rev_set:
+        rev_set.difference_update(branch_names.keys())
 
     exact_name = repo.experiments.get_exact_name(rev_set)
 
     for baseline, baseline_results in result.items():
+        name_set: Set[str] = set()
         for rev, rev_result in baseline_results.items():
             name: Optional[str] = None
             if rev == "baseline":
                 rev = baseline
                 if rev == "workspace":
                     continue
-            name = names.get(rev, None) or exact_name[rev]
-            if name:
+            name = branch_names.get(rev, None) or exact_name[rev]
+            if name and name not in name_set:
+                name_set.add(name)
                 rev_result["data"]["name"] = name
 
 
@@ -336,6 +354,40 @@ def update_new(
             to_dict[baseline][rev] = to_dict[baseline].get(rev, experiment)
 
 
+def move_properties_to_head(result: Dict[str, Dict[str, Dict[str, Any]]]):
+    for _, baseline_results in result.items():
+        checkpoint: bool = False
+        head: Dict[str, Any] = {}
+        for rev, rev_data in baseline_results.items():
+            if (
+                "data" not in rev_data
+                or rev_data["data"].get("checkpoint_tip", None) is None
+            ):
+                checkpoint = False
+                head = {}
+                continue
+
+            rev_result: Dict[str, Any] = rev_data["data"]
+            if (
+                checkpoint is True
+                and rev_result["checkpoint_tip"] == head["checkpoint_tip"]
+            ):
+                if "name" in rev_result and "name" not in head:
+                    head["name"] = rev_result["name"]
+                    del rev_result["name"]
+                if rev_result["executor"]:
+                    if not head["executor"]:
+                        head["executor"] = rev_result["executor"]
+                    rev_result["executor"] = None
+                if rev_result["status"] == ExpStatus.Running.name:
+                    head["status"] = ExpStatus.Running.name
+                    rev_result["status"] = ExpStatus.Success.name
+            else:
+                if rev_result["checkpoint_tip"] == rev:
+                    head = rev_result
+                    checkpoint = True
+
+
 def show(
     repo: "Repo",
     all_branches=False,
@@ -354,8 +406,7 @@ def show(
     if repo.scm.no_commits:
         return {}
 
-    if onerror is None:
-        onerror = _show_onerror_collect
+    onerror = onerror or _show_onerror_collect
 
     res: Dict[str, Dict] = defaultdict(OrderedDict)
 
@@ -368,6 +419,7 @@ def show(
     found_revs.update(
         iter_revs(repo.scm, revs, num, all_branches, all_tags, all_commits)
     )
+    branch_names = get_branch_names(repo.scm, found_revs)
 
     running: Dict[str, Dict] = repo.experiments.get_running_exps(
         fetch_refs=fetch_running
@@ -421,6 +473,8 @@ def show(
     update_new(res, queued_experiment)
 
     if not sha_only:
-        get_names(repo, res)
+        update_names(repo, branch_names, res)
+
+    move_properties_to_head(res)
 
     return res


Test patch:
diff --git a/tests/unit/repo/experiments/test_show.py b/tests/unit/repo/experiments/test_show.py
--- a/tests/unit/repo/experiments/test_show.py
+++ b/tests/unit/repo/experiments/test_show.py
@@ -1,7 +1,13 @@
+import pytest
 from funcy import first
 from scmrepo.git import Git
 
-from dvc.repo.experiments.show import get_names
+from dvc.repo.experiments.show import (
+    ExpStatus,
+    get_branch_names,
+    move_properties_to_head,
+    update_names,
+)
 
 
 def test_get_show_branch(tmp_dir, scm: "Git", dvc, exp_stage):
@@ -21,7 +27,192 @@ def test_get_show_branch(tmp_dir, scm: "Git", dvc, exp_stage):
         baseline: {"baseline": {"data": {}}, exp_a: {"data": {}}},
     }
 
-    get_names(dvc, result)
+    branch_names = get_branch_names(scm, ["workspace", exp_a, baseline])
+    assert branch_names == {exp_a: new_branch, baseline: base_branch}
+
+    update_names(dvc, branch_names, result)
     assert result[exp_a]["baseline"]["data"] == {"name": new_branch}
     assert result[baseline]["baseline"]["data"] == {"name": base_branch}
     assert result[baseline][exp_a]["data"] == {"name": new_branch}
+
+
+@pytest.mark.parametrize(
+    "res_in,res_out",
+    [
+        (
+            {
+                "1": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "2": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "a",
+                    }
+                },
+                "3": {
+                    "data": {
+                        "checkpoint_tip": "3",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "4": {
+                    "data": {
+                        "checkpoint_tip": "3",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "b",
+                    }
+                },
+            },
+            {
+                "1": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "a",
+                    }
+                },
+                "2": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "3": {
+                    "data": {
+                        "checkpoint_tip": "3",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "b",
+                    }
+                },
+                "4": {
+                    "data": {
+                        "checkpoint_tip": "3",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+            },
+        ),
+        (
+            {
+                "0": {"error": {"msg": "something"}},
+                "1": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "2": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "a",
+                    }
+                },
+                "3": {"data": {"name": "b"}},
+            },
+            {
+                "0": {"error": {"msg": "something"}},
+                "1": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "a",
+                    }
+                },
+                "2": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "3": {"data": {"name": "b"}},
+            },
+        ),
+        (
+            {
+                "1": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "2": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "a",
+                    }
+                },
+                "3": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "b",
+                    }
+                },
+                "4": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                    }
+                },
+            },
+            {
+                "1": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": "local",
+                        "status": ExpStatus.Running.name,
+                        "name": "a",
+                    }
+                },
+                "2": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+                "3": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                        "name": "b",
+                    }
+                },
+                "4": {
+                    "data": {
+                        "checkpoint_tip": "1",
+                        "executor": None,
+                        "status": ExpStatus.Success.name,
+                    }
+                },
+            },
+        ),
+    ],
+)
+def test_move_properties_to_head(res_in, res_out):
+    move_properties_to_head({"baseline": res_in})
+    assert res_in == res_out