Repository: iterative/dvc

Issue:
`exp show`: experiment order randomly changing
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

<!--
A clear and concise description of what the bug is.
-->

When calling `exp show` multiple times the order of experiments can change between calls (no sorts applied):


https://github.com/iterative/dvc/assets/37993418/7ca95c89-907a-438c-ae10-343c19c9f50d



### Reproduce

<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

1. queue several experiments
2. start the queue with multiple workers
3. monitor exp show

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

Experiments order is static unless sorts are applied.

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.58.1 (pip)
-------------------------
Platform: Python 3.10.6 on macOS-13.3.1-arm64-arm-64bit
Subprojects:
        dvc_data = 0.51.0
        dvc_objects = 0.22.0
        dvc_render = 0.5.0
        dvc_task = 0.2.1
        scmrepo = 1.0.3
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.5.0, boto3 = 1.26.76)
Config:
        Global: /Users/mattseddon/Library/Application Support/dvc
        System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/9b2d1355ef26cb2a38a704c54d1f3710
```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->

Noticed while integrating https://github.com/iterative/dvc/pull/9432 with the extension (for https://github.com/iterative/vscode-dvc/issues/3436 & https://github.com/iterative/vscode-dvc/issues/3178)


Discussion:
Discussed this in slack, the issue is probably due to (lack of) git commit timestamp precision. The default sort is chronological by git timestamp, there may be a way to get more consistent ordering but I don't think it needs to be prioritized

Resolving patch:
diff --git a/dvc/repo/experiments/collect.py b/dvc/repo/experiments/collect.py
--- a/dvc/repo/experiments/collect.py
+++ b/dvc/repo/experiments/collect.py
@@ -10,6 +10,7 @@
     Iterator,
     List,
     Optional,
+    Tuple,
     Union,
 )
 
@@ -369,12 +370,13 @@ def _describe(
 
 
 def _sorted_ranges(exp_ranges: Iterable["ExpRange"]) -> List["ExpRange"]:
-    """Return list of ExpRange sorted by timestamp."""
+    """Return list of ExpRange sorted by (timestamp, rev)."""
 
-    def _head_timestamp(exp_range: "ExpRange") -> datetime:
+    def _head_timestamp(exp_range: "ExpRange") -> Tuple[datetime, str]:
         head_exp = first(exp_range.revs)
         if head_exp and head_exp.data and head_exp.data.timestamp:
-            return head_exp.data.timestamp
-        return datetime.fromtimestamp(0)
+            return head_exp.data.timestamp, head_exp.rev
+
+        return datetime.fromtimestamp(0), ""
 
     return sorted(exp_ranges, key=_head_timestamp, reverse=True)


Test patch:
diff --git a/tests/unit/repo/experiments/test_collect.py b/tests/unit/repo/experiments/test_collect.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/repo/experiments/test_collect.py
@@ -0,0 +1,69 @@
+import datetime
+import random
+from typing import Dict, List
+
+import pytest
+
+from dvc.repo.experiments.collect import ExpRange, ExpState, SerializableExp, collect
+
+
+@pytest.mark.vscode
+def test_collect_stable_sorting(dvc, scm, mocker):
+    """
+    Check that output is deterministically sorted even for
+    commits with the same timestamp. This affects the experience
+    in vs-code to avoid experiments "bouncing around" when "exp show"
+    is called repeatedly
+    """
+    expected_revs = [
+        "c" * 40,
+        "b" * 40,
+        "a" * 40,
+        "7" * 40,
+    ]
+
+    def collect_queued_patched(_, baseline_revs) -> Dict[str, List["ExpRange"]]:
+        single_timestamp = datetime.datetime(2023, 6, 20, 0, 0, 0)
+
+        exp_ranges = [
+            ExpRange(
+                revs=[
+                    ExpState(
+                        rev=rev,
+                        name=f"exp-state-{rev[0]}",
+                        data=SerializableExp(rev=rev, timestamp=single_timestamp),
+                    )
+                ],
+                name=f"exp-range-{rev[0]}",
+            )
+            for rev in expected_revs
+        ]
+
+        # shuffle collection order
+        random.shuffle(exp_ranges)
+
+        return {baseline_rev: exp_ranges for baseline_rev in baseline_revs}
+
+    mocker.patch("dvc.repo.experiments.collect.collect_queued", collect_queued_patched)
+    mocker.patch("dvc.repo.experiments.collect.collect_active", return_value={})
+    mocker.patch("dvc.repo.experiments.collect.collect_failed", return_value={})
+    mocker.patch("dvc.repo.experiments.collect.collect_successful", return_value={})
+
+    # repeat (shuffling collection order in collect_queued_patched)
+    for _ in range(20):
+        collected = collect(repo=dvc, all_commits=True)
+        assert collected[0].rev == "workspace"
+        assert collected[0].experiments is None
+        assert collected[1].rev == scm.get_rev()
+        _assert_experiment_rev_order(collected[1].experiments, expected_revs)
+
+
+def _assert_experiment_rev_order(
+    actual: List["ExpRange"],
+    expected_revs: List[str],
+):
+    expected_revs = expected_revs.copy()
+
+    for actual_exp_range in actual:
+        for exp_state in actual_exp_range.revs:
+            assert exp_state.rev == expected_revs.pop(0)