Repository: cocotb/cocotb

Issue:
Potential rerun/reproducibility problem when using TESTCASE with multi-test MODULE
Opening PR per @ktbarrett in discussion here: https://gitter.im/cocotb/Lobby

In short, cocotb sets random.seed at the start of running a MODULE in init.py but does not set the seed again at the start of each @cocotb.test().  If the MODULE had multiple tests in it and a user sets TESTCASE to rerun any test but the first one in MODULE, then it's possible the random behavior for the rerun test will be different than the original failure.

A few thoughts:   Resetting the seed back to RANDOM_SEED (or the randomly generated seed) before each test would not be an attractive solution because then the random behavior for each test could be identical for all MODULE tests.   Ideally, the "testcase seed" would be set to a different value for each test and derived from/based on the RANDOM_SEED (or randomly generated seed).   One way to get such a seed could be to offset the main seed based on some algorithm based on the test currently running or in other words:   Testcase Seed = Main Seed + **f(testcase)**        If this were true then:

- Each test would get a unique seed hopefully producing different random behavior than the other tests
- When rerunning a failing test in a multi-test MODULE, the user would only need to set RANDOM_SEED to the same Main Seed which was used (likely randomly generated) for the original failure.

I am new to cocotb so I could be missing details but it appears MODULE test names are essentially unique in that cocotb seems to combine all @cocotb.test() names in the MODULE which are the same into one test, no matter where they appear in the file.  If so, then one possibility for the **f(testcase)** offset would be to hash the testcase name to produce a unique number.  Granted, it's possible the same number might result from different test names but would be hopefully uncommon.




Discussion:
Each `cocotb.test()` decorated function gets a unique `_id` that could be used, but setting `MODULE` to a list of modules would change things depending on the order.



> Each `cocotb.test()` decorated function gets a unique `_id` that could be used, but setting `MODULE` to a list of modules would change things depending on the order.

Interesting, I did not know that - thx!  I looked at decorators.py where I think the _id you are referring to is assigned and if so, it appears _id is an incrementally increasing number (+1) for each test.  In my experience, it's not uncommon for a verification methodology to extract and store failing tests for regression purposes.  If the _id was used to offset the seed and the failing test was not the first one in the MODULE file, then I think a user could not clean out the other passing tests in the MODULE file and only save the failing one(s) for future regressions.   It seems like it would be best if the seed offset used to create the testcase seed was not related to order/position in the original MODULE file. 


I think the best solution would be to set a seed per test when the test is created. If you maintain the same MODULE in the same order (and you should, test modules can have side effects, changing the modules loaded or the order can break things) then each time the RM loads the tests, they get the same seed based off the RANDOM_SEED. From that point running a particular test will run with that same seed.
An alternative might be to assign a test seed based on the RANDOM_SEED and the test name (could be done as simply as XORing the RANDOM_SEED with the hash of the test name). Test names (with the module name) are unique. This would allow you to add or remove tests and have the same RANDOM_SEED work as long as the test isn't renamed or the test body changed.
Yes, that seems to work.  When we found this, we locally fixed the problem in `regression.py` by importing `hashlib` and adding the following code at line 302. We just offset cocotb.RANDOM_SEED but perhaps XORing would be better.

       ```
        h = hashlib.new('shake_128')
        h.update(test.__qualname__.encode())
        offset=int((h.hexdigest(4)),16)
        testcase_seed=cocotb.RANDOM_SEED+offset
        random.seed(testcase_seed)
        self.log.debug("Seeding testcase=%s cocotb.RANDOM_SEED=%s Offset=%s Testcase_seed=%s" % (test.__qualname__,cocotb.RANDOM_SEED,offset,testcase_seed))
```

One issue with using the name of the test is that tests generated by `TestFactory` are fragile. Add more values to a `TestFactory` option and the indexes shift. It's another argument for computing the name of `TestFactory`-generated tests based on unique representations of the argument value, like pytest does.

I don't think using the test name to set a test seed on creation breaks any existing reasonable interfaces, so I'll try to get this in soon.

Resolving patch:
diff --git a/cocotb/regression.py b/cocotb/regression.py
--- a/cocotb/regression.py
+++ b/cocotb/regression.py
@@ -36,6 +36,8 @@
 import pdb
 from typing import Any, Optional, Tuple, Iterable
 from functools import wraps
+import random
+import hashlib
 
 import cocotb
 import cocotb.ANSI as ANSI
@@ -308,8 +310,16 @@ def _init_test(self, test: Test) -> Optional[RunningTask]:
             self._record_result(test, test_init_outcome, 0, 0)
             return None
 
-        test = test_init_outcome.get()
-        return test
+        running_test = test_init_outcome.get()
+
+        # seed random number generator based on test module, name, and RANDOM_SEED
+        hasher = hashlib.sha1()
+        hasher.update(test.__qualname__.encode())
+        hasher.update(test.__module__.encode())
+        seed = cocotb.RANDOM_SEED + int(hasher.hexdigest(), 16)
+        random.seed(seed)
+
+        return running_test
 
     def _score_test(self, test: Test, outcome: Outcome) -> Tuple[bool, bool]:
         """


Test patch:
diff --git a/tests/test_cases/test_seed/.gitignore b/tests/test_cases/test_seed/.gitignore
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_seed/.gitignore
@@ -0,0 +1 @@
+number
diff --git a/tests/test_cases/test_seed/Makefile b/tests/test_cases/test_seed/Makefile
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_seed/Makefile
@@ -0,0 +1,10 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+
+run:
+	rm -f number
+	$(MAKE) sim MODULE=test_other,test_seed RANDOM_SEED=1234
+	$(MAKE) sim MODULE=test_seed TESTCASE=test_reproducibility RANDOM_SEED=1234
+
+include ../../designs/sample_module/Makefile
diff --git a/tests/test_cases/test_seed/test_other.py b/tests/test_cases/test_seed/test_other.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_seed/test_other.py
@@ -0,0 +1,11 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+import cocotb
+
+
+@cocotb.test()
+async def test_pass(_):
+    # exists solely so there is another test in another module
+    # before the other module is run
+    pass
diff --git a/tests/test_cases/test_seed/test_seed.py b/tests/test_cases/test_seed/test_seed.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cases/test_seed/test_seed.py
@@ -0,0 +1,25 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+import random
+
+import cocotb
+
+
+@cocotb.test()
+async def test_first(_):
+    # move generator to test that it doesn't affect the next test
+    for _ in range(100):
+        random.getrandbits(64)
+
+
+@cocotb.test()
+async def test_reproducibility(_):
+    try:
+        with open("number") as file:
+            a = int(file.read())
+        assert a == random.getrandbits(32)
+    except FileNotFoundError:
+        with open("number", "w") as file:
+            number = random.getrandbits(32)
+            file.write(str(number))