Repository: cocotb/cocotb

Issue:
Function for comparing simulator version strings?
Icarus Verilog master, reporting ``SIM_VERSION`` "11.0 (devel)", now supports bit selects (see https://github.com/steveicarus/iverilog/issues/323).
Thus, we need a way to selectively expect an error in our tests based on the version.

While I could compare with either the "11.0" string part, or the "10." string of the released version (and assume nobody uses a version before 10), or treat the number part as a float etc., this gets all ugly for the general case.
Since we have to support more such older/newer comparisons in the future (e.g. for upcoming GHDL improvements), does it make sense to provide a version comparison function that would gradually be extended to handle more simulators as needed?


Discussion:
It is a good idea to encapsulate this into a function, but it should probably be multiple functions, one per simulator, and another function to map the SIM_NAME to a simulator. But how would the version comparison function work? Would it "normalize" a version? Would you feed it a partial version for it to match? Would it support version comparison?
Can we use `distutils.version.LooseVersion` here?
Indeed, http://epydoc.sourceforge.net/stdlib/distutils.version.LooseVersion-class.html looks like a good start, we probably don't even need to encapsulate this further for now, only when pathological cases show up.
> we probably don't even need to encapsulate this further for now, only when pathological cases show up

I'd err on the side of caution here. We have a chance to prevent a large refactor in the future. Do all SIM_VERSION strings for all simulators start with the main version? Do some of them contain preambles? How does `LooseVersion` handle version strings like Questa's where they started using both versioning schemes before switching from semver to calver (e.g. `10.7b 2018.06`)? It's a simple as making otherwise empty aliases for now:

```python
class IcarusVersion(LooseVersion):
    pass
```

I think at some point it would be nice to co-locate all the simulator-specific information used by cocotb, and this might be good place to start.
Interestingly, ``LooseVersion`` doesn't like Aldec Active-HDL version numbers (taken from https://github.com/VUnit/vunit/pull/585/files#diff-8e7169ad9a827f7ebe5cb68193d01c63R423-R437):
```
Failed example:
    ActivehdlVersion("10.5a.12.6914") > ActivehdlVersion("10.5.216.6767")
Exception raised:
    Traceback (most recent call last):
      File "/tools/anaconda/2019.10-python3.7/Linux-x86_64/lib/python3.7/doctest.py", line 1329, in __run
        compileflags, 1), test.globs)
      File "<doctest __main__.ActivehdlVersion[0]>", line 1, in <module>
        ActivehdlVersion("10.5a.12.6914") > ActivehdlVersion("10.5.216.6767")
      File "/tools/anaconda/2019.10-python3.7/Linux-x86_64/lib/python3.7/distutils/version.py", line 64, in __gt__
        c = self._cmp(other)
      File "/tools/anaconda/2019.10-python3.7/Linux-x86_64/lib/python3.7/distutils/version.py", line 337, in _cmp
        if self.version < other.version:
    TypeError: '<' not supported between instances of 'str' and 'int'
```
This bug is still present in Python 3.9-dev: https://travis-ci.org/github/cocotb/cocotb/jobs/707016738
That's unfortunate, thanks for trying it.
We could give `LooseVersion` some help here, that's why I suggested wrapper classes. But I'm not quite sure I understand the versioning system Active-HDL uses. The example you posted makes no sense to me.
This may be fixed in https://github.com/python/cpython/pull/14952, but that doesn't help us much.
> The example you posted makes no sense to me.

~It could be a synthetic example that doesn't occur in the wild.~
Apparently it is a real version: https://github.com/VUnit/vunit/issues/584#issuecomment-552767043
https://github.com/python/cpython/blob/master/Lib/distutils/version.py#L260-L262 - ha!
...OTOH: https://github.com/python/cpython/blob/master/Lib/distutils/version.py#L296-L297
Inheriting from `Version` might still be the right answer. We can embed knowledge about the versioning in a particular simulator to reject invalid version numbers.
I guess I'll make this not a doctest but a regular docstring with a note.
With luck, if we ever need this, we won't have problematic version numbers.

Resolving patch:
diff --git a/cocotb/_sim_versions.py b/cocotb/_sim_versions.py
new file mode 100644
--- /dev/null
+++ b/cocotb/_sim_versions.py
@@ -0,0 +1,119 @@
+# Copyright cocotb contributors
+# Licensed under the Revised BSD License, see LICENSE for details.
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""
+Classes to compare simulation versions.
+
+These are for cocotb-internal use only.
+
+.. warning::
+    These classes silently allow comparing versions of different simulators.
+"""
+
+from distutils.version import LooseVersion
+
+
+class ActivehdlVersion(LooseVersion):
+    """Version numbering class for Aldec Active-HDL.
+
+    NOTE: unsupported versions exist, e.g.
+    ActivehdlVersion("10.5a.12.6914") > ActivehdlVersion("10.5.216.6767")
+    """
+    pass
+
+
+class CvcVersion(LooseVersion):
+    """Version numbering class for Tachyon DA CVC.
+
+    Example:
+        >>> CvcVersion("OSS_CVC_7.00b-x86_64-rhel6x of 07/07/14 (Linux-elf)") > CvcVersion("OSS_CVC_7.00a-x86_64-rhel6x of 07/07/14 (Linux-elf)")
+        True
+    """
+    pass
+
+
+class GhdlVersion(LooseVersion):
+    """Version numbering class for GHDL."""
+    pass
+
+
+class IcarusVersion(LooseVersion):
+    """Version numbering class for Icarus Verilog.
+
+    Example:
+        >>> IcarusVersion("11.0 (devel)") > IcarusVersion("10.3 (stable)")
+        True
+        >>> IcarusVersion("10.3 (stable)") <= IcarusVersion("10.3 (stable)")
+        True
+    """
+    pass
+
+
+class ModelsimVersion(LooseVersion):
+    """Version numbering class for Mentor ModelSim."""
+    pass
+
+
+class QuestaVersion(LooseVersion):
+    """Version numbering class for Mentor Questa.
+
+    Example:
+        >>> QuestaVersion("10.7c 2018.08") > QuestaVersion("10.7b 2018.06")
+        True
+        >>> QuestaVersion("2020.1 2020.01") > QuestaVersion("10.7c 2018.08")
+        True
+    """
+    pass
+
+
+class RivieraVersion(LooseVersion):
+    """Version numbering class for Aldec Riviera-PRO.
+
+    Example:
+        >>> RivieraVersion("2019.10.138.7537") == RivieraVersion("2019.10.138.7537")
+        True
+    """
+    pass
+
+
+class VcsVersion(LooseVersion):
+    """Version numbering class for Synopsys VCS.
+
+    Example:
+        >>> VcsVersion("Q-2020.03-1_Full64") > VcsVersion("K-2015.09_Full64")
+        True
+    """
+    pass
+
+
+class VerilatorVersion(LooseVersion):
+    """Version numbering class for Verilator.
+
+    Example:
+        >>> VerilatorVersion("4.032 2020-04-04") > VerilatorVersion("4.031 devel")
+        True
+    """
+    pass
+
+
+class XceliumVersion(LooseVersion):
+    """Version numbering class for Cadence Xcelium.
+
+    Example:
+        >>> XceliumVersion("20.06-g183") > XceliumVersion("20.03-s002")
+        True
+        >>> XceliumVersion("20.07-e501") > XceliumVersion("20.06-g183")
+        True
+    """
+    pass
+
+
+class IusVersion(XceliumVersion):  # inherit everything from Xcelium
+    """Version numbering class for Cadence IUS.
+
+    Example:
+        >>> IusVersion("15.20-s050") > IusVersion("15.20-s049")
+        True
+    """
+    pass


Test patch:
diff --git a/tests/test_cases/test_discovery/test_discovery.py b/tests/test_cases/test_discovery/test_discovery.py
--- a/tests/test_cases/test_discovery/test_discovery.py
+++ b/tests/test_cases/test_discovery/test_discovery.py
@@ -30,6 +30,7 @@
 from cocotb.triggers import Timer
 from cocotb.result import TestError, TestFailure
 from cocotb.handle import IntegerObject, ConstantObject, HierarchyObject, StringObject
+from cocotb._sim_versions import IcarusVersion
 
 
 @cocotb.test()
@@ -89,8 +90,8 @@ async def access_signal(dut):
 
 
 @cocotb.test(
-    # Icarus 10.3 doesn't support bit-selects, see https://github.com/steveicarus/iverilog/issues/323
-    expect_error=IndexError if cocotb.SIM_NAME.lower().startswith("icarus") else False,
+    # Icarus up to (including) 10.3 doesn't support bit-selects, see https://github.com/steveicarus/iverilog/issues/323
+    expect_error=IndexError if (cocotb.SIM_NAME.lower().startswith("icarus") and (IcarusVersion(cocotb.SIM_VERSION) <= IcarusVersion("10.3 (stable)"))) else False,
     skip=cocotb.LANGUAGE in ["vhdl"])
 async def access_single_bit(dut):
     """Access a single bit in a vector of the DUT"""
@@ -107,8 +108,8 @@ async def access_single_bit(dut):
 
 
 @cocotb.test(
-    # Icarus 10.3 doesn't support bit-selects, see https://github.com/steveicarus/iverilog/issues/323
-    expect_error=IndexError if cocotb.SIM_NAME.lower().startswith("icarus") else False,
+    # Icarus up to (including) 10.3 doesn't support bit-selects, see https://github.com/steveicarus/iverilog/issues/323
+    expect_error=IndexError if (cocotb.SIM_NAME.lower().startswith("icarus") and (IcarusVersion(cocotb.SIM_VERSION) <= IcarusVersion("10.3 (stable)"))) else False,
     skip=cocotb.LANGUAGE in ["vhdl"])
 async def access_single_bit_assignment(dut):
     """Access a single bit in a vector of the DUT using the assignment mechanism"""