Repository: pytest-dev/pytest-randomly (language: python)

Issue:
Incompatible with `pytest-mypy-plugins` (3.10 version)
### Python Version

3.8.11

### Package Version

3.10

### Description

I am trying to update `pytest-randomly` from `3.8` to `3.10`. But my test suite now fails:

```
 ============================= test session starts ==============================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
Using --randomly-seed=2912292964
rootdir: /home/runner/work/returns/returns, configfile: setup.cfg
plugins: randomly-3.10.0, subtests-0.5.0, xdist-2.3.0, hypothesis-6.14.6, returns-0.16.0, mypy-plugins-1.7.0, anyio-3.3.0, forked-1.3.0
collected 830 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/_pytest/main.py", line 637, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
INTERNALERROR>     self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 212, in pytest_collection_modifyitems
INTERNALERROR>     _shuffle_by_class(list(group), seed),
INTERNALERROR>   File "/home/runner/work/returns/returns/.venv/lib/python3.8/site-packages/pytest_randomly/__init__.py", line 239, in _shuffle_by_class
INTERNALERROR>     klass_items.sort()
INTERNALERROR> TypeError: '<' not supported between instances of 'YamlTestItem' and 'YamlTestItem'
```

Link: https://github.com/dry-python/returns/pull/1021/checks?check_run_id=3322840792

I guess this happens because we use [`pytest-mypy-plugins`](https://github.com/typeddjango/pytest-mypy-plugins) where we define our tests in `yml` files, example: https://github.com/dry-python/returns/blob/master/typesafety/test_functions/test_tap.yml

Steps to reproduce: 
1. Clone https://github.com/dry-python/returns
2. Run `poetry install`
3. Run `poetry run pip install 'pytest-randomly==3.10.0'`
4. Run `poetry run pytest typesafety -p no:cov -o addopts="" --mypy-ini-file=setup.cfg
`: https://github.com/dry-python/returns/blob/master/.github/workflows/test.yml#L67


Discussion / hints:

Fix patch:
diff --git a/HISTORY.rst b/HISTORY.rst
index ed2e3b7..16d0d99 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -2,6 +2,12 @@
 History
 =======
 
+* Fix new shuffling to work when one or more test in a class or module have the
+  same test id.
+
+  Thanks to Nikita Sobolev for the report in `Issue #378
+  <https://github.com/pytest-dev/pytest-randomly/issues/378>`__.
+
 3.10.0 (2021-08-13)
 -------------------
 


Test patch:
diff --git a/src/pytest_randomly/__init__.py b/src/pytest_randomly/__init__.py
index 9f98f3a..9f4501e 100644
--- a/src/pytest_randomly/__init__.py
+++ b/src/pytest_randomly/__init__.py
@@ -234,15 +234,13 @@ def _get_module(item: Item) -> Optional[ModuleType]:
 def _shuffle_by_class(items: List[Item], seed: int) -> List[Item]:
     klasses_items: List[Tuple[Optional[Type[Any]], List[Item]]] = []
 
+    def _item_key(item: Item) -> bytes:
+        return _md5(f"{seed}::{item.nodeid}")
+
     for klass, group in groupby(items, _get_cls):
-        klass_items = [(_md5(f"{seed}::{item.nodeid}"), item) for item in group]
-        klass_items.sort()
-        klasses_items.append(
-            (
-                klass,
-                [item for _key, item in klass_items],
-            )
-        )
+        klass_items = list(group)
+        klass_items.sort(key=_item_key)
+        klasses_items.append((klass, klass_items))
 
     def _cls_key(klass_items: Tuple[Optional[Type[Any]], List[Item]]) -> bytes:
         klass, items = klass_items
diff --git a/tests/test_pytest_randomly.py b/tests/test_pytest_randomly.py
index 2178597..3ad44e4 100644
--- a/tests/test_pytest_randomly.py
+++ b/tests/test_pytest_randomly.py
@@ -463,6 +463,11 @@ def pytest_collect_file(path, parent):
                 parent=parent,
                 fspath=str(path),
                 items=[
+                    NoOpItem.from_parent(
+                        name=str(path) + "1",
+                        parent=parent,
+                        module=sys.modules[__name__],
+                    ),
                     NoOpItem.from_parent(
                         name=str(path) + "1",
                         parent=parent,
@@ -479,7 +484,7 @@ def pytest_collect_file(path, parent):
     args = ["-v"]
 
     out = ourtestdir.runpytest(*args)
-    out.assert_outcomes(passed=2)
+    out.assert_outcomes(passed=3)
 
 
 def test_doctests_in_txt_files_reordered(ourtestdir):