Repository: pytest-dev/pytest-xdist
Pull request #796 (2022-07-16 07:48:37)

Added known limitation section to the docs
- added known limitations to documentation

Thanks for submitting a PR, your contribution is really appreciated!

Here's a quick checklist that should be present in PRs:

- [x] Make sure to include reasonable tests for your change if necessary

- [x] We use [towncrier](https://pypi.python.org/pypi/towncrier) for changelog management, so please add a *news* file into the `changelog` folder following these guidelines:
  * Name it `$issue_id.$type` for example `588.bugfix`;
  * If you don't have an issue_id change it to the PR id after creating it
  * Ensure type is one of `removal`, `feature`, `bugfix`, `vendor`, `doc` or `trivial`
  * Make sure to use full sentences with correct case and punctuation, for example:

    ```
    Fix issue with non-ascii contents in doctest text files.
    ```


RonnyPfannschmidt: Im a bit torn between limitations ynd design constraints, but nicely done 
07pepa: i missed changes in testing..
07pepa: @nicoddemus i tried to implement all your sugestion and om some of them i expanded 

Patch:
diff --git a/changelog/796.doc.rst b/changelog/796.doc.rst
new file mode 100644
index 0000000..28a10bc
--- /dev/null
+++ b/changelog/796.doc.rst
@@ -0,0 +1,1 @@
+Added known limitations section to documentation.
diff --git a/docs/index.rst b/docs/index.rst
index 94747b1..6e8e611 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -60,4 +60,5 @@ Features
     crash
     how-to
     how-it-works
+    known-limitations
     changelog
diff --git a/docs/known-limitations.rst b/docs/known-limitations.rst
new file mode 100644
index 0000000..04641a7
--- /dev/null
+++ b/docs/known-limitations.rst
@@ -0,0 +1,52 @@
+Known limitations
+=================
+
+pytest-xdist has some limitations that may be supported in pytest but can't be supported in pytest-xdist.
+
+Order and amount of test must be consistent
+-------------------------------------------
+
+Is is not possible to have tests that differ in order or their amount across workers.
+
+This is especially true with ``pytest.mark.parametrize``, when values are produced with sets or other unordered iterables/generators.
+
+
+Example:
+
+.. code-block:: python
+
+    import pytest
+
+    @pytest.mark.parametrize("param", {"a","b"})
+    def test_pytest_parametrize_unordered(param):
+        pass
+
+In the example above, the fact that ``set`` are not necessarily ordered can cause different workers
+to collect tests in different order, which will throw an error.
+
+Workarounds
+~~~~~~~~~~~
+
+A solution to this is to guarantee that the parametrized values have the same order.
+
+Some solutions:
+
+* Convert your sequence to a ``list``.
+
+  .. code-block:: python
+
+    import pytest
+
+    @pytest.mark.parametrize("param", ["a", "b"])
+    def test_pytest_parametrize_unordered(param):
+        pass
+
+* Sort your sequence, guaranteeing order.
+
+  .. code-block:: python
+
+    import pytest
+
+    @pytest.mark.parametrize("param", sorted({"a", "b"}))
+    def test_pytest_parametrize_unordered(param):
+        pass
diff --git a/src/xdist/report.py b/src/xdist/report.py
index 02ad30d..d956577 100644
--- a/src/xdist/report.py
+++ b/src/xdist/report.py
@@ -14,7 +14,8 @@ def report_collection_diff(from_collection, to_collection, from_id, to_id):
     error_message = (
         "Different tests were collected between {from_id} and {to_id}. "
         "The difference is:\n"
-        "{diff}"
+        "{diff}\n"
+        "To see why this happens see Known limitations in documentation"
     ).format(from_id=from_id, to_id=to_id, diff="\n".join(diff))
     msg = "\n".join(x.rstrip() for x in error_message.split("\n"))
     return msg


Test patch:
diff --git a/testing/test_dsession.py b/testing/test_dsession.py
index 464045e..d3a5715 100644
--- a/testing/test_dsession.py
+++ b/testing/test_dsession.py
@@ -290,7 +290,8 @@ def test_report_collection_diff_different() -> None:
         " bbb\n"
         "+XXX\n"
         " ccc\n"
-        "-YYY"
+        "-YYY\n"
+        "To see why this happens see Known limitations in documentation"
     )
 
     msg = report_collection_diff(from_collection, to_collection, "1", "2")