Repository: huggingface/transformers
Pull request: Add pytest-random-order for reproducible test randomization

# What does this PR do?

Uses bucket=module to randomize tests within files, compatible with --dist=loadfile. CI uses CIRCLE_BUILD_NUM as seed for reproducibility across parallel containers. Local runs use random seed to catch order dependencies over time. Enables debugging by reproducing exact test order from failed CI builds.
<!-- Remove if not applicable -->


Changed files:
filename: .circleci/create_circleci_config.py
patch: @@ -32,7 +32,7 @@
     "DISABLE_SAFETENSORS_CONVERSION": True,
 }
 # Disable the use of {"s": None} as the output is way too long, causing the navigation on CircleCI impractical
-COMMON_PYTEST_OPTIONS = {"max-worker-restart": 0, "vvv": None, "rsfE":None}
+COMMON_PYTEST_OPTIONS = {"max-worker-restart": 0, "vvv": None, "rsfE":None, "random-order-bucket": "module", "random-order-seed": "${CIRCLE_BUILD_NUM:-0}"}
 DEFAULT_DOCKER_IMAGE = [{"image": "cimg/python:3.8.12"}]
 
 # Strings that commonly appear in the output of flaky tests when they fail. These are used with `pytest-rerunfailures`
@@ -111,6 +111,8 @@ def __post_init__(self):
             self.install_steps = ["uv pip install ."]
         # Use a custom patched pytest to force exit the process at the end, to avoid `Too long with no output (exceeded 10m0s): context deadline exceeded`
         self.install_steps.append("uv pip install git+https://github.com/ydshieh/pytest.git@8.4.1-ydshieh")
+        # Install pytest-random-order plugin for test randomization
+        self.install_steps.append("uv pip install pytest-random-order")
         if self.pytest_options is None:
             self.pytest_options = {}
         if isinstance(self.tests_to_run, str):

filename: Makefile
patch: @@ -59,13 +59,13 @@ fix-repo: style
 	-python utils/add_dates.py
 
 
-# Run tests for the library
+# Run tests for the library, requires pytest-random-order
 test:
-	python -m pytest -n auto --dist=loadfile -s -v ./tests/
+	python -m pytest -p random_order -n auto --dist=loadfile -s -v --random-order-bucket=module ./tests/
 
-# Run tests for examples
+# Run tests for examples, requires pytest-random-order
 test-examples:
-	python -m pytest -n auto --dist=loadfile -s -v ./examples/pytorch/
+	python -m pytest -p random_order -n auto --dist=loadfile -s -v --random-order-bucket=module ./examples/pytorch/
 
 # Run benchmark
 benchmark:

filename: docs/source/en/testing.md
patch: @@ -327,14 +327,13 @@ There is another plugin `pytest-repeat`, but it doesn't work with `unittest`.
 
 #### Run tests in a random order
 
+The transformers project uses [`pytest-random-order`](https://github.com/jbasko/pytest-random-order) to randomize test execution order. This is included in the dev dependencies:
+
 ```bash
 pip install pytest-random-order
 ```
 
-Important: the presence of `pytest-random-order` will automatically randomize tests, no configuration change or
-command line options is required.
-
-As explained earlier this allows detection of coupled tests - where one test's state affects the state of another. When
+Test randomization allows detection of coupled tests - where one test's state affects the state of another. When
 `pytest-random-order` is installed it will print the random seed it used for that session, e.g:
 
 ```bash
@@ -344,36 +343,41 @@ Using --random-order-bucket=module
 Using --random-order-seed=573663
 ```
 
-So that if the given particular sequence fails, you can reproduce it by adding that exact seed, e.g.:
+**Bucket Modes:** pytest-random-order supports different bucket modes that control the granularity of randomization:
+- `module` (default in this project): Randomizes tests within each file, but keeps files in consistent order. Works well with `--dist=loadfile` for parallel execution.
+- `class`: Randomizes within each test class
+- `package`: Randomizes within each package
+- `global`: Randomizes all tests globally
 
-```bash
-pytest --random-order-seed=573663
-[...]
-Using --random-order-bucket=module
-Using --random-order-seed=573663
-```
+**In CircleCI:** Tests automatically use a deterministic seed based on the build number (`CIRCLE_BUILD_NUM`), ensuring:
+- Different test order for each build/PR (catches order dependencies over time)
+- Same order across all parallel containers in the same build (reproducible failures)
+- Ability to reproduce failures locally
+- bucket=module mode ensures predictable load balancing with pytest-xdist
+
+**Locally with `make test`:** Tests use `--random-order-bucket=module` with a random seed each run.
 
-It will only reproduce the exact order if you use the exact same list of tests (or no list at all). Once you start to
-manually narrowing down the list you can no longer rely on the seed, but have to list them manually in the exact order
-they failed and tell pytest to not randomize them instead using `--random-order-bucket=none`, e.g.:
+**Reproducing a specific test order:** If a particular sequence fails in CI, you can reproduce it locally using the build number:
 
 ```bash
-pytest --random-order-bucket=none tests/test_a.py tests/test_c.py tests/test_b.py
+pytest --random-order-bucket=module --random-order-seed=12345
 ```
 
-To disable the shuffling for all tests:
+The seed can be found in the CircleCI build environment variables or test output.
+
+**To disable randomization** for a single run:
 
 ```bash
 pytest --random-order-bucket=none
 ```
 
-By default `--random-order-bucket=module` is implied, which will shuffle the files on the module levels. It can also
-shuffle on `class`, `package`, `global` and `none` levels. For the complete details please see its
-[documentation](https://github.com/jbasko/pytest-random-order).
+Or set a fixed seed:
+
+```bash
+pytest --random-order-seed=0
+```
 
-Another randomization alternative is: [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly). This
-module has a very similar functionality/interface, but it doesn't have the bucket modes available in
-`pytest-random-order`. It has the same problem of imposing itself once installed.
+**Alternative:** [`pytest-randomly`](https://github.com/pytest-dev/pytest-randomly) is another option that also sets random seeds in test code (random.seed, numpy.random.seed), but lacks bucket modes for controlling randomization granularity.
 
 ### Look and feel variations
 

filename: setup.py
patch: @@ -111,6 +111,7 @@
     "pydantic>=2",
     "pytest>=7.2.0,<9.0.0",
     "pytest-asyncio>=1.2.0",
+    "pytest-random-order",
     "pytest-rerunfailures<16.0",
     "pytest-timeout",
     "pytest-xdist",
@@ -270,6 +271,7 @@ def run(self):
     deps_list(
         "pytest",
         "pytest-asyncio",
+        "pytest-random-order",
         "pytest-rich",
         "pytest-xdist",
         "pytest-order",

filename: src/transformers/dependency_versions_table.py
patch: @@ -49,6 +49,7 @@
     "pydantic": "pydantic>=2",
     "pytest": "pytest>=7.2.0,<9.0.0",
     "pytest-asyncio": "pytest-asyncio>=1.2.0",
+    "pytest-random-order": "pytest-random-order",
     "pytest-rerunfailures": "pytest-rerunfailures<16.0",
     "pytest-timeout": "pytest-timeout",
     "pytest-xdist": "pytest-xdist",