Repository: pytest-dev/pytest-xdist
Pull request #766 (2022-03-21 15:02:38)

Added a couple tests for #404
As explained in https://github.com/pytest-dev/pytest-xdist/issues/404#issuecomment-1073998712


Patch:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index ad3fd32..9c2836d 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
 -   repo: https://github.com/psf/black
-    rev: 22.1.0
+    rev: 22.3.0
     hooks:
     -   id: black
         args: [--safe, --quiet, --target-version, py35]
@@ -10,7 +10,7 @@ repos:
     -   id: blacken-docs
         additional_dependencies: [black==20.8b1]
 -   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.1.0
+    rev: v4.2.0
     hooks:
     -   id: trailing-whitespace
     -   id: end-of-file-fixer
@@ -21,7 +21,7 @@ repos:
     hooks:
     -   id: flake8
 -   repo: https://github.com/asottile/pyupgrade
-    rev: v2.31.1
+    rev: v2.32.1
     hooks:
     -   id: pyupgrade
         args: [--py3-plus]
@@ -34,7 +34,7 @@ repos:
         language: python
         additional_dependencies: [pygments, restructuredtext_lint]
 -   repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.940
+    rev: v0.960
     hooks:
     -   id: mypy
         files: ^(src/|testing/)
diff --git a/changelog/791.doc b/changelog/791.doc
new file mode 100644
index 0000000..adc3e65
--- /dev/null
+++ b/changelog/791.doc
@@ -0,0 +1,1 @@
+Document the ``pytest_xdist_auto_num_workers`` hook.
diff --git a/changelog/796.doc.rst b/changelog/796.doc.rst
new file mode 100644
index 0000000..28a10bc
--- /dev/null
+++ b/changelog/796.doc.rst
@@ -0,0 +1,1 @@
+Added known limitations section to documentation.
diff --git a/docs/distribution.rst b/docs/distribution.rst
index 5a86743..0ec9416 100644
--- a/docs/distribution.rst
+++ b/docs/distribution.rst
@@ -5,12 +5,24 @@ Running tests across multiple CPUs
 
 To send tests to multiple CPUs, use the ``-n`` (or ``--numprocesses``) option::
 
-    pytest -n 8
+    pytest -n auto
 
-Pass ``-n auto`` to use as many processes as your computer has CPU cores. This
-can lead to considerable speed ups, especially if your test suite takes a
+This can lead to considerable speed ups, especially if your test suite takes a
 noticeable amount of time.
 
+With ``-n auto``, pytest-xdist will use as many processes as your computer
+has CPU cores.
+Pass a number, e.g. ``-n 8``, to specify the number of processes explicitly.
+
+To specify a different meaning for ``-n auto`` for your tests,
+you can implement the ``pytest_xdist_auto_num_workers``
+`pytest hook <https://docs.pytest.org/en/latest/how-to/writing_plugins.html>`__
+(a function named ``pytest_xdist_auto_num_workers`` in e.g. ``conftest.py``)
+that returns the number of processes to use.
+
+
+Parallelization can be configured further with these options:
+
 * ``--maxprocesses=maxprocesses``: limit the maximum number of workers to
   process the tests.
 
diff --git a/docs/index.rst b/docs/index.rst
index 94747b1..6e8e611 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -60,4 +60,5 @@ Features
     crash
     how-to
     how-it-works
+    known-limitations
     changelog
diff --git a/docs/known-limitations.rst b/docs/known-limitations.rst
new file mode 100644
index 0000000..04641a7
--- /dev/null
+++ b/docs/known-limitations.rst
@@ -0,0 +1,52 @@
+Known limitations
+=================
+
+pytest-xdist has some limitations that may be supported in pytest but can't be supported in pytest-xdist.
+
+Order and amount of test must be consistent
+-------------------------------------------
+
+Is is not possible to have tests that differ in order or their amount across workers.
+
+This is especially true with ``pytest.mark.parametrize``, when values are produced with sets or other unordered iterables/generators.
+
+
+Example:
+
+.. code-block:: python
+
+    import pytest
+
+    @pytest.mark.parametrize("param", {"a","b"})
+    def test_pytest_parametrize_unordered(param):
+        pass
+
+In the example above, the fact that ``set`` are not necessarily ordered can cause different workers
+to collect tests in different order, which will throw an error.
+
+Workarounds
+~~~~~~~~~~~
+
+A solution to this is to guarantee that the parametrized values have the same order.
+
+Some solutions:
+
+* Convert your sequence to a ``list``.
+
+  .. code-block:: python
+
+    import pytest
+
+    @pytest.mark.parametrize("param", ["a", "b"])
+    def test_pytest_parametrize_unordered(param):
+        pass
+
+* Sort your sequence, guaranteeing order.
+
+  .. code-block:: python
+
+    import pytest
+
+    @pytest.mark.parametrize("param", sorted({"a", "b"}))
+    def test_pytest_parametrize_unordered(param):
+        pass
diff --git a/docs/remote.rst b/docs/remote.rst
index 7ffab18..997bef3 100644
--- a/docs/remote.rst
+++ b/docs/remote.rst
@@ -10,7 +10,7 @@ tests that you can successfully run locally. And you
 have a ssh-reachable machine ``myhost``.  Then
 you can ad-hoc distribute your tests by typing::
 
-    pytest -d --tx ssh=myhostpopen --rsyncdir mypkg mypkg
+    pytest -d  --rsyncdir mypkg --tx ssh=myhostpopen mypkg/tests/unit/test_something.py
 
 This will synchronize your :code:`mypkg` package directory
 to a remote ssh account and then locally collect tests
@@ -49,7 +49,7 @@ It will tell you that it starts listening on the default
 port.  You can now on your home machine specify this
 new socket host with something like this::
 
-    pytest -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg mypkg
+    pytest -d --tx socket=192.168.1.102:8888 --rsyncdir mypkg
 
 
 
diff --git a/setup.cfg b/setup.cfg
index 03e4a91..4994681 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -45,7 +45,7 @@ install_requires =
     execnet>=1.1
     pytest>=6.2.0
     pytest-forked
-setup_requires = setuptools_scm>=6.0
+setup_requires = # left empty, enforce using isolated build system
 
 [options.packages.find]
 where = src
diff --git a/src/xdist/report.py b/src/xdist/report.py
index 02ad30d..d956577 100644
--- a/src/xdist/report.py
+++ b/src/xdist/report.py
@@ -14,7 +14,8 @@ def report_collection_diff(from_collection, to_collection, from_id, to_id):
     error_message = (
         "Different tests were collected between {from_id} and {to_id}. "
         "The difference is:\n"
-        "{diff}"
+        "{diff}\n"
+        "To see why this happens see Known limitations in documentation"
     ).format(from_id=from_id, to_id=to_id, diff="\n".join(diff))
     msg = "\n".join(x.rstrip() for x in error_message.split("\n"))
     return msg
diff --git a/tox.ini b/tox.ini
index 2578ae5..0c1655b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -5,7 +5,7 @@ envlist=
   py38-pytestmain
   py38-psutil
   py38-setproctitle
-
+isolated_build = true
 [testenv]
 extras = testing
 deps =


Test patch:
diff --git a/testing/test_dsession.py b/testing/test_dsession.py
index 464045e..d3a5715 100644
--- a/testing/test_dsession.py
+++ b/testing/test_dsession.py
@@ -290,7 +290,8 @@ def test_report_collection_diff_different() -> None:
         " bbb\n"
         "+XXX\n"
         " ccc\n"
-        "-YYY"
+        "-YYY\n"
+        "To see why this happens see Known limitations in documentation"
     )
 
     msg = report_collection_diff(from_collection, to_collection, "1", "2")
diff --git a/testing/test_workermanage.py b/testing/test_workermanage.py
index fae0601..b19c524 100644
--- a/testing/test_workermanage.py
+++ b/testing/test_workermanage.py
@@ -3,9 +3,12 @@ import py
 import pytest
 import shutil
 import textwrap
+import warnings
 from pathlib import Path
+from util import generate_warning
 from xdist import workermanage
-from xdist.workermanage import HostRSync, NodeManager
+from xdist.remote import serialize_warning_message
+from xdist.workermanage import HostRSync, NodeManager, unserialize_warning_message
 
 pytest_plugins = "pytester"
 
@@ -326,3 +329,80 @@ class TestNodeManager:
         )
         (rep,) = reprec.getreports("pytest_runtest_logreport")
         assert rep.passed
+
+
+class MyWarning(UserWarning):
+    pass
+
+
+@pytest.mark.parametrize(
+    "w_cls",
+    [
+        UserWarning,
+        MyWarning,
+        "Imported",
+        pytest.param(
+            "Nested",
+            marks=pytest.mark.xfail(reason="Nested warning classes are not supported."),
+        ),
+    ],
+)
+def test_unserialize_warning_msg(w_cls):
+    """Test that warning serialization process works well"""
+
+    # Create a test warning message
+    with pytest.warns(UserWarning) as w:
+        if not isinstance(w_cls, str):
+            warnings.warn("hello", w_cls)
+        elif w_cls == "Imported":
+            generate_warning()
+        elif w_cls == "Nested":
+            # dynamic creation
+            class MyWarning2(UserWarning):
+                pass
+
+            warnings.warn("hello", MyWarning2)
+
+    # Unpack
+    assert len(w) == 1
+    w_msg = w[0]
+
+    # Serialize and deserialize
+    data = serialize_warning_message(w_msg)
+    w_msg2 = unserialize_warning_message(data)
+
+    # Compare the two objects
+    all_keys = set(vars(w_msg).keys()).union(set(vars(w_msg2).keys()))
+    for k in all_keys:
+        v1 = getattr(w_msg, k)
+        v2 = getattr(w_msg2, k)
+        if k == "message":
+            assert type(v1) == type(v2)
+            assert v1.args == v2.args
+        else:
+            assert v1 == v2
+
+
+class MyWarningUnknown(UserWarning):
+    # Changing the __module__ attribute is only safe if class can be imported
+    # from there
+    __module__ = "unknown"
+
+
+def test_warning_serialization_tweaked_module():
+    """Test for GH#404"""
+
+    # Create a test warning message
+    with pytest.warns(UserWarning) as w:
+        warnings.warn("hello", MyWarningUnknown)
+
+    # Unpack
+    assert len(w) == 1
+    w_msg = w[0]
+
+    # Serialize and deserialize
+    data = serialize_warning_message(w_msg)
+
+    # __module__ cannot be found!
+    with pytest.raises(ModuleNotFoundError):
+        unserialize_warning_message(data)
diff --git a/testing/util.py b/testing/util.py
new file mode 100644
index 0000000..c7bcc55
--- /dev/null
+++ b/testing/util.py
@@ -0,0 +1,9 @@
+import warnings
+
+
+class MyWarning2(UserWarning):
+    pass
+
+
+def generate_warning():
+    warnings.warn(MyWarning2("hello"))