Repository: Qiskit/qiskit

Issue:
Non-deterministic test failures in CI `test_layout_tokyo`
### Environment

- **Qiskit Terra version**: main @ a3188f5
- **Python version**: any
- **Operating system**: any

### What is happening?

I've seen the same test failure on CI for Windows with Python 3.7 on two different PRs now: #7902 and #7995.  The CI logs look like:
```text
test.python.transpiler.test_preset_passmanagers.TestFinalLayouts.test_layout_tokyo_2845_2_1
-------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "D:\a\1\s\test-job\lib\site-packages\ddt.py", line 191, in wrapper
    return func(self, *args, **kwargs)

      File "D:\a\1\s\test\python\transpiler\test_preset_passmanagers.py", line 593, in test_layout_tokyo_2845
    self.assertEqual(result._layout._p2v, expected_layouts[level])

      File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\unittest\case.py", line 852, in assertEqual
    assertion_func(first, second, msg=msg)

      File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\unittest\case.py", line 1151, in assertDictEqual
    self.fail(self._formatMessage(msg, standardMsg))

      File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\unittest\case.py", line 693, in fail
    raise self.failureException(msg)

    AssertionError: {11: Qubit(QuantumRegister(3, 'qr1'), 1), [843 chars] 14)} != {0: Qubit(QuantumRegister(15, 'ancilla'), [843 chars] 14)}
Diff is 1572 characters long. Set self.maxDiff to None to see it.
```

### How can we reproduce the issue?

Run CI, but it seems to be non-deterministic.  I've seen the same test case go through after a restart as well.

### What should happen?

No test failure, and certainly it shouldn't be stochastic.

### Any suggestions?

Both times I saw it, the failing test was Windows with Python 3.7 and `TestFinalLayouts.test_layout_tokyo_2845_2_1`, which I _think_ is `optimization_level=1` (`ddt` is doing something weird with the naming), but that might be a red herring.  It's possible that the transpiler seed isn't getting passed down to Rust completely, or that some dependency has changed for us, affecting the random-number generation.

*edit*: I've now seen this on #7995 on macOS with Python 3.10, so it's not limited to Windows and 3.7.


Discussion:
I'm wondering if this is related to #7862, that added a new heuristic scoring to adjust the layout after routing and I could see the score having some we floating point precision sensitivity (which I always think of when I see windows failures like this).
This isn't just limited to Windows - #7995 has had the same failure on the macOS Python 3.10 run.
So this is definitely being caused by the `VF2PostLayout` pass looking at the azure test analytics the first failure occured during devlopment in #7862 and it's only failed since that merged. Talking this over with @jakelishman I think the source of the failure is the `time_limit` parameter on the pass. If we get a particularly slow CI node the test is likely hitting the time limit before the pass finds the expected layout and is returning whatever the best layout it's found so far is when the time limit is hit (which is different from the expected result). The `time_limit` value for optimization level 1 is 100ms which is small enough that a slow CI node could take too long to find the expected layout.

We can confirm this by enabling debug logging on that test as the `VF2PostLayout` pass logs the score for each mapping returned by retworkx and when it finds a better mapping it logs the layout too. Although maybe we should just bump the timeout in the test to ensure we have more of a margin.

Resolving patch:
diff --git a/qiskit/transpiler/preset_passmanagers/level1.py b/qiskit/transpiler/preset_passmanagers/level1.py
--- a/qiskit/transpiler/preset_passmanagers/level1.py
+++ b/qiskit/transpiler/preset_passmanagers/level1.py
@@ -156,7 +156,6 @@ def _vf2_match_not_found(property_set):
             coupling_map,
             seed=seed_transpiler,
             call_limit=int(5e4),  # Set call limit to ~100ms with retworkx 0.10.2
-            time_limit=0.1,
             properties=backend_properties,
             target=target,
         )
@@ -325,7 +324,6 @@ def _post_layout_condition(property_set):
                     backend_properties,
                     seed_transpiler,
                     call_limit=int(5e4),  # Set call limit to ~100ms with retworkx 0.10.2
-                    time_limit=0.1,
                     strict_direction=False,
                 ),
                 condition=_trivial_not_perfect,
diff --git a/qiskit/transpiler/preset_passmanagers/level2.py b/qiskit/transpiler/preset_passmanagers/level2.py
--- a/qiskit/transpiler/preset_passmanagers/level2.py
+++ b/qiskit/transpiler/preset_passmanagers/level2.py
@@ -154,7 +154,6 @@ def _vf2_match_not_found(property_set):
             coupling_map,
             seed=seed_transpiler,
             call_limit=int(5e6),  # Set call limit to ~10 sec with retworkx 0.10.2
-            time_limit=10.0,
             properties=backend_properties,
             target=target,
         )
@@ -312,7 +311,6 @@ def _post_layout_condition(property_set):
                     backend_properties,
                     seed_transpiler,
                     call_limit=int(5e6),  # Set call limit to ~10 sec with retworkx 0.10.2
-                    time_limit=10.0,
                     strict_direction=False,
                 )
             )
diff --git a/qiskit/transpiler/preset_passmanagers/level3.py b/qiskit/transpiler/preset_passmanagers/level3.py
--- a/qiskit/transpiler/preset_passmanagers/level3.py
+++ b/qiskit/transpiler/preset_passmanagers/level3.py
@@ -157,7 +157,6 @@ def _vf2_match_not_found(property_set):
             coupling_map,
             seed=seed_transpiler,
             call_limit=int(3e7),  # Set call limit to ~60 sec with retworkx 0.10.2
-            time_limit=60,
             properties=backend_properties,
             target=target,
         )
@@ -323,7 +322,6 @@ def _post_layout_condition(property_set):
                     backend_properties,
                     seed_transpiler,
                     call_limit=int(3e7),  # Set call limit to ~60 sec with retworkx 0.10.2
-                    time_limit=60,
                     strict_direction=False,
                 )
             )


Test patch: