Repository: cocotb/cocotb

Issue:
"Unable set up RisingEdge(c) Trigger" with Cadence IUS/Xcelium and VHDL
Current status: See https://github.com/cocotb/cocotb/issues/1076#issuecomment-1351266296 for a summary of the problem.

Cadence tracks the implementation of the missing functionality as Jira ticket AVSREQ-183427.

---

When I run ``examples/dff/tests`` with 
```
make SIM=ius TOPLEVEL_LANG=vhdl COCOTB_LOG_LEVEL=DEBUG COCOTB_SCHEDULER_DEBUG=1 all
```
I am getting the following output:
```
     0.00ns DEBUG    cocotb.regression                         regression.py:555  in generate_tests                  Adding generated test "run_test_001" to module "dff_cocotb"
     0.00ns INFO     cocotb.regression                         regression.py:187  in initialise                      Found test dff_cocotb.run_test_001
     0.00ns INFO     cocotb.regression                         regression.py:321  in execute                         Running test 1/1: run_test_001
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:618  in add                             Adding new coroutine run_test_001
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine run_test_001 with outcome Value(None)
     0.00ns INFO     ...test.run_test_001.0x7f4ae5046780       decorators.py:253  in _advance                        Starting test: "run_test_001"
                                                                                                                     Description: Automatically generated test
                                                                                                                     
                                                                                                                     
     0.00ns DEBUG    ...test.run_test_001.0x7f4ae5046780       decorators.py:258  in _advance                        Sending Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine run_test_001 yielded run_test (mode 1)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:645  in _trigger_from_unstarted_coro    Scheduling nested coroutine: run_test
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:618  in add                             Adding new coroutine run_test
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine run_test with outcome Value(None)
     0.00ns DEBUG    cocotb.c                                      handle.py:84   in __init__                        Created
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:618  in add                             Adding new coroutine start
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine start with outcome Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _do_writes with outcome Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine _do_writes yielded _Event (mode 1)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:372  in _event_loop                     Trigger fired: _Event
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:434  in _event_loop                     1 pending coroutines for event _Event
                                                                                                                     	_do_writes
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:441  in _event_loop                     Scheduling coroutine _do_writes
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _do_writes with outcome Value(<cocotb.triggers._Event object at 0x7f4ae491f548>)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine _do_writes yielded ReadWrite(readwritesync) (mode 1)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:444  in _event_loop                     Scheduled coroutine _do_writes
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:463  in _event_loop                     All coroutines scheduled, handing control back to simulator
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine start yielded Timer(2500000.00ps) (mode 1)
     0.00ns DEBUG    cocotb.d                                      handle.py:84   in __init__                        Created
     0.00ns DEBUG    cocotb.q                                      handle.py:84   in __init__                        Created
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:618  in add                             Adding new coroutine _monitor_recv
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _monitor_recv with outcome Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine _monitor_recv yielded RisingEdge(c) (mode 1)
     0.00ns ERROR    cocotb.RisingEdge.0x7f4ae4925c28              result.py:52   in raise_error                     Unable set up RisingEdge(c) Trigger
                                                                                                                     NoneType: None
                                                                                                                     
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:372  in _event_loop                     Trigger fired: NullTrigger()
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:434  in _event_loop                     1 pending coroutines for event NullTrigger()
                                                                                                                     	_monitor_recv
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:441  in _event_loop                     Scheduling coroutine _monitor_recv
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _monitor_recv with outcome Error(TestError('Unable set up RisingEdge(c) Trigger'))
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:714  in schedule                        Coroutine completed: _monitor_recv
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:491  in unschedule                      TestComplete received: TestError
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:754  in finish_test                     finish_test called with TestError('Unable set up RisingEdge(c) Trigger')
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:780  in cleanup                         Killing start
     0.00ns DEBUG    ..tb.coroutine.start.0x7f4ae4924198       decorators.py:182  in kill                            kill() called on coroutine
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:780  in cleanup                         Killing _do_writes
     0.00ns DEBUG    ..routine._do_writes.0x7f4ae4924278       decorators.py:182  in kill                            kill() called on coroutine
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:780  in cleanup                         Killing run_test_001
     0.00ns DEBUG    ...test.run_test_001.0x7f4ae5046780       decorators.py:182  in kill                            kill() called on coroutine
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:444  in _event_loop                     Scheduled coroutine _monitor_recv
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:276  in _check_termination              Test terminating, scheduling Timer
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:463  in _event_loop                     All coroutines scheduled, handing control back to simulator
     0.00ns INFO     cocotb.scoreboard.dff                     scoreboard.py:216  in add_interface                   Created with reorder_depth 0
     0.00ns DEBUG    cocotb.monitor.BitMonitor                   __init__.py:112  in add_callback                    Adding callback of function check_received_transaction to monitor
     0.00ns DEBUG    cocotb.monitor.BitMonitor                   __init__.py:112  in add_callback                    Adding callback of function model to monitor
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:618  in add                             Adding new coroutine _monitor_recv
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _monitor_recv with outcome Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine _monitor_recv yielded RisingEdge(c) (mode 4)
     0.00ns ERROR    cocotb.RisingEdge.0x7f4ae4925c28              result.py:52   in raise_error                     Unable set up RisingEdge(c) Trigger
                                                                                                                     NoneType: None
                                                                                                                     
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:372  in _event_loop                     Trigger fired: NullTrigger()
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:378  in _event_loop                     Ignoring trigger NullTrigger() since we're terminating
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:618  in add                             Adding new coroutine _cr_twiddler
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _cr_twiddler with outcome Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:695  in schedule                        Scheduling coroutine _do_writes with outcome Value(None)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine _do_writes yielded _Event (mode 4)
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:372  in _event_loop                     Trigger fired: _Event
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:378  in _event_loop                     Ignoring trigger _Event since we're terminating
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine _cr_twiddler yielded RisingEdge(c) (mode 4)
     0.00ns ERROR    cocotb.RisingEdge.0x7f4ae4925c28              result.py:52   in raise_error                     Unable set up RisingEdge(c) Trigger
                                                                                                                     NoneType: None
                                                                                                                     
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:372  in _event_loop                     Trigger fired: NullTrigger()
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:378  in _event_loop                     Ignoring trigger NullTrigger() since we're terminating
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:701  in schedule                        Coroutine run_test yielded RisingEdge(c) (mode 4)
     0.00ns ERROR    cocotb.RisingEdge.0x7f4ae4925c28              result.py:52   in raise_error                     Unable set up RisingEdge(c) Trigger
                                                                                                                     NoneType: None
                                                                                                                     
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:372  in _event_loop                     Trigger fired: NullTrigger()
     0.00ns DEBUG    cocotb.scheduler                           scheduler.py:378  in _event_loop                     Ignoring trigger NullTrigger() since we're terminating
     1.00ns DEBUG    cocotb.scheduler                           scheduler.py:303  in begin_test                      begin_test called with trigger: Timer(1000.00ps)
     1.00ns DEBUG    cocotb.scheduler                           scheduler.py:319  in begin_test                      Issue test result to regression object
     1.00ns ERROR    cocotb.regression                         regression.py:300  in handle_result                   Test Failed: run_test_001 (result was TestError)
     1.00ns ERROR    cocotb.regression                         regression.py:207  in tear_down                       Failed 1 out of 1 tests (0 skipped)
```
(I have modified line 695 of ``scheduler.py`` to read ``self.log.debug("Scheduling coroutine {} with outcome {}".format(coroutine.__name__, send_outcome))`` here.)

Any hints as to how to approach debugging this?


Discussion:
For some reason `simulator.register_value_change_callback` is failing, and unfortunately it doesn't propagate a reason
I think if you checkout master (or some commit containing #985), you'll get tracebacks pointing to that line?
I am on master, and there is no traceback. In fact, I found that I do not get any of the ``LOG_ERROR`` messages - if I replace them with ``fprintf(stderr, ...)``, I get something.
The code is taking this path: https://github.com/cocotb/cocotb/blob/master/cocotb/share/lib/vhpi/VhpiCbHdl.cpp#L745,
and from ``arm_callback`` I am getting the message "VHPI ERROR: Registered callback isn't enabled! Got 1000".
This in turn is explained by the message from the simulator when run with ``-plidebug``:
```
ERROR:          VHPI       VHUNIMP
         The property or method vhpiStateP has not been implemented yet.
```
So far so bad.
The ``arm_callback`` method for VPI looks a bit different, maybe that logic could also be applied to VHPI...
>  In fact, I found that I do not get any of the LOG_ERROR messages - if I replace them with fprintf(stderr, ...), I get something.

That's a little alarming - perhaps `LOG_ERROR` is hanging trying to acquire the GIL. Alternatively, it might be outputting to the simulator console, if you're running in gui mode.

>The property or method vhpiStateP has not been implemented yet.

Is this #153 again? The cause looks like these lines:

https://github.com/cocotb/cocotb/blob/ddcd4b27ddaa9dd96b6cfe8bcaf390be0d4befe8/cocotb/share/lib/vhpi/VhpiCbHdl.cpp#L409-L413


It's not outputting to the simulator console.
And good find, I had forgotten about this ancient issue. The conversation doesn't seem to be very helpful now unfortunately, and my bug report to Cadence that resulted from it went unanswered.
Logging output is fixed in gh-1086
I believe this is the same problem as issue #381. I am running into this same issue with IUS.
FWIW, in current master (and thus the upcoming version 1.5) this is printed slightly different, i.e.:
```
0.00ns ERROR    cocotb.gpi                                VhpiCbHdl.cpp:448  in arm_callback                    VHPI: Registered callback isn't enabled! Got vhpiStateP=vhpiUndefined(1000)
```
We're encountering this issue when using Xcelium too. Because it supports cross-language compilation, you can write a Verilog shim. This makes cocotb talk to the simulator over VPI which doesn't encounter similar issues as the VHPI codepath.

## Workaround example
<details>
<summary>Click to expand</summary>

The following cocotb test case + hdl files work fine in GHDL but fail with Xcelium:

**adder.vhdl**:
```vhdl
port
  ( clk : in std_logic
  ; a : in signed(31 downto 0)
  ; b : in signed(31 downto 0)
  ; c : out signed(31 downto 0)
  );
end adder;

architecture Behavioral of adder is
begin
  p_add : process(clk)
  begin
    if rising_edge(clk) then
      c <= a + b;
    end if;
  end process p_add;
end Behavioral;
```

**test.py**:

```python
import cocotb

from cocotb.clock import Clock
from cocotb.triggers import RisingEdge

@cocotb.test()
async def simple_test(dut):
    clock = Clock(dut.clk, 50, units='ns')
    cocotb.fork(clock.start())

    dut.a.value = 5
    dut.b.value = 7

    # Wait for simulator to start
    await RisingEdge(dut.clk)

    # Latch a+b
    await RisingEdge(dut.clk)

    # a+b on output
    await RisingEdge(dut.clk)

    assert dut.c.value.integer == 12
```

**Makefile**:
```makefile
SIM ?= ghdl
TOPLEVEL_LANG ?= vhdl
MODULE := test

VHDL_SOURCES = adder.vhdl

include $(shell cocotb-config --makefiles)/Makefile.sim
```

To work around this, add a Verilog shim:

**adder_shim.v**:
```verilog
module adder_shim
  ( input         clk
  , input  [31:0] a
  , input  [31:0] b
  , output [31:0] c
  );

  adder d0 (clk, a, b, c);
endmodule
```

and change your `Makefile` such that it uses the shim when compiling with  Xcelium:

```makefile
SIM ?= ghdl
TOPLEVEL_LANG ?= vhdl
MODULE := test

VHDL_SOURCES = adder.vhdl

ifneq ($(filter $(SIM),xcelium),)
    VERILOG_SOURCES = adder_shim.v
    TOPLEVEL_LANG := verilog
    TOPLEVEL := adder_shim
else
    TOPLEVEL_LANG := vhdl
    TOPLEVEL := adder
endif

include $(shell cocotb-config --makefiles)/Makefile.sim
```

</details>
After having implemented a way to automate the shim generation, this issue came back to bite us again while trying to schedule triggers on signals deep inside a design. We settled on a few functions that can act as a "poor man"'s implementation of `RisingEdge`/`FallingEdge` that polls for an edge based on a reference clock:

```python
async def poll_edge(poll_trigger, signal, edge_left, edge_right):
    """Internal function for `poll_rising_edge` and `poll_falling_edge`"""
    if len(signal) != 1:
        raise ValueError(f"`signal` should be exactly one bit wide, not {len(signal)}.")

    value_prev = signal.value.integer
    while True:
        await poll_trigger 
        value_curr = signal.value.integer
        if value_prev == edge_left and value_curr == edge_right:
            break
        value_prev = value_curr

async def poll_rising_edge(poll_trigger, signal):
    """
    Poll a signal for a rising edge at every `poll_trigger`.

    This is poor-man's implementation of `RisingEdge(signal)` used to workaround simulator
    schedueling issues described in https://github.com/cocotb/cocotb/issues/1076. 

    Example usage:

    > await poll_rising_edge(RisingEdge(dut.main_clock), dut.some.signal)
    """
    await poll_edge(poll_trigger, signal, 0, 1)

async def poll_falling_edge(poll_trigger, signal):
    """
    Poll a signal for a falling edge at every `poll_trigger`.

    This is poor-man's implementation of `RisingEdge(signal)` used to workaround simulator
    schedueling issues described in https://github.com/cocotb/cocotb/issues/1076. 

    Example usage:

    > await poll_falling_edge(RisingEdge(dut.main_clock), dut.some.signal)
    """
    await poll_edge(poll_trigger, signal, 1, 0)
```
To add a bit more context here for people coming to this bug:

- **User-visible problem:** cocotb users with VHDL toplevel designs cannot use RisingEdge() triggers in Xcelium and IUS.
- **Workaround:** Use a Verilog toplevel.
- **Summary of the underlying issue:** Cadence Xcelium and IUS do not support a VHPI feature cocotb relies on to implement RisingEdge triggers. 

## Steps to reproduce the issue

```
$ git clone https://github.com/cocotb/cocotb.git
$ python3 -m venv cocotb-venv
$ cd cocotb-venv
$ . bin/activate
$ pip3 install cocotb==1.7.2 pytest
$ make -C ../cocotb/examples/simple_dff SIM=xcelium TOPLEVEL_LANG=vhdl COCOTB_LOG_LEVEL=DEBUG COCOTB_SCHEDULER_DEBUG=1 all
```

The output will include the following lines:
```
     0.00ns ERROR    gpi                                VHPI: Registered callback isn't enabled! Got vhpiStateP=vhpiUndefined(1000)
     0.00ns ERROR    gpi                                Failed to register a value change callback
 ```

where Xcelium indicates that it does not support the `vhpiStateP` property of `vhpi_get()`.

## Analysis

The calling code in question is `vhpi_get(vhpiStateP, new_hdl);` at https://github.com/cocotb/cocotb/blob/ac41bea02b9e09a617fca95421f36dac5029d247/cocotb/share/lib/vhpi/VhpiCbHdl.cpp#L452-L458

`vhpi_get()` obtains the property of an object, in this case of `new_hdl`, a callback handle. `vhpiStateP` indicates which type of property we'd like to get: the state of the callback. `vhpi_get()` should return one of three options: `vhpiEnable`, `vhpiDisable` or `vhpiMature`. Instead, Xcelium returns `vhpiUndefined`, indicating that it does not support getting the `vhpiStateP` property from a callback handle.

It should be noted that using `vhpiStateP` is not really obscure, as seen in the example code in the [VHDL IEEE Std 1076-2019 spec](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8938196) section "23.8 vhpi_enable_cb":


> In the following VHPI program, the function vhpi_enable_cb is used to enable a callback that was registered but is disabled.
```cpp
static vhpiHandleT mylastcbk = 0;

void activate_cbk(vhpiHandle cbHdl) {
  vhpiStateT cbState;
  cbState = vhpi_get(vhpiStateP, cbHdl);
  if (cbState == vhpiDisable)
    vhpi_enable_cb(cbHdl);
  }
}

void register_cbk() {
  vhpiCbDataT cbData;
  vhpiHandleT cbHdl;
  int flags;
  flags = vhpiDisableCb | vhpiReturnCb;
  cbData.reason = vhpiCbEndOfSimulation;
  cbData.cb_rtn = myf;
  cbHdl = vhpi_register_cb(&cbData, flags);
  mylastcbk = cbHdl;
}

int main (int argc, char *argv[] ){
  register_cbk();
  ...
  activate_cbk(mylastcbk);
  return(0);
}
```
Probably a stupid question, since I have no insight into VHPI, but is there no other way the RisingEdge trigger could be implemented than via the current approach?
We always register callbacks as enabled, so we could assume the state is `vhpiEnable` without checking it.
I tried removing the check, and the simulation gets started. However, it later fails with the following message:

50.00ns  ERROR  gpi  FATAL: We are calling up again

This is with Xcelium 18.12
You can try setting `COCOTB_LOG_LEVEL` environment variable to `DEBUG` or `TRACE` to get more info, but they will be increasingly verbose, so if a lot happens before that 50ns message you could get a flood of log entries.

You could also attempt to debug with gdb using https://docs.cocotb.org/en/stable/building.html#envvar-COCOTB_ATTACH
The DUT seems to be correctly discovered, but then this happens:

![image](https://user-images.githubusercontent.com/282215/213050440-151e43fa-816b-4db6-8738-c4fa70cbcc8e.png)

I wonder whether this memory corruption error has anything to do with skipping the checks earlier. Seems to coincide (in simulation time) with the first await on RisingEdge.
This looks like a case of callback re-entrancy as detailed in #2178.

The TRACE entry "Passing control to GPI user" happens when the simulator calls a registered GPI callback.
That is happening during another callback.

If you could debug it and put a break point in `handle_vhpi_callback()`, you might get some better insight by inspecting the `vhpiCbDataT` struct.
I managed to attach GDB and set the breakpoint - however, it seems there is no debug information available in the loaded cocotb libraries, so I can't do "info args", for example. I tried adding "-g" to _extra_cxx_compile_args in cocotb_build_libs.py and do a reinstall with "pip3 --user --no-build-isolation .", but it didn't help.
If you have `nox` installed, the `dev` session installs and editable build with debug info and coverage.

Run `nox -s dev -- bash` from the cocotb repo. That will setup the environment and run bash with everything installed. You can also use a bash alternative.

I've confirmed with Icarus Verilog that the cb_data info is available:

```gdb
(gdb) b handle_vpi_callback
Breakpoint 1 at 0x7f5c6b948100: file cocotb/share/lib/vpi/VpiImpl.cpp, line 629.
(gdb) continue
Continuing.

Breakpoint 1, handle_vpi_callback (cb_data=0x55e859fadcd0) at cocotb/share/lib/vpi/VpiImpl.cpp:629
629     int32_t handle_vpi_callback(p_cb_data cb_data) {
(gdb) where
#0  handle_vpi_callback (cb_data=0x55e859fadcd0) at cocotb/share/lib/vpi/VpiImpl.cpp:629
#1  0x000055e8596e848b in vpiStartOfSim () at vpi_callback.cc:541
#2  0x000055e8596b3e11 in schedule_simulate () at schedule.cc:1154
#3  0x000055e8596649a5 in main (argc=<optimized out>, argv=<optimized out>) at main.cc:469
(gdb) print *cb_data
$2 = {reason = 11, cb_rtn = 0x7f5c6b948100 <handle_vpi_callback(p_cb_data)>, obj = 0x0, time = 0x55e859f2f2e0,
  value = 0x0, index = 0, user_data = 0x55e859f2f2a0 "@_\227k\\\177"}
```
Ok, I got cocotb compiled with debug info on. Not using nox (since I'm behind a corporate firewall, and downloading packages is a pain), but instead invoking pip as follows:

`pip3 install --user --no-build-isolation --global-option build --global-option --debug .`

Then I attached  GDB and ran the following session:

```
(gdb) b handle_vhpi_callback
Breakpoint 1 at 0x7f9136e84a6d: file cocotb/share/lib/vhpi/VhpiImpl.cpp, line 988.

(gdb) handle SIGSEGV nostop noprint pass
Signal        Stop      Print   Pass to program Description
SIGSEGV       No        No      Yes             Segmentation fault

(gdb) cont
Continuing.
warning: Corrupted shared library list: 0x578d9f0 != 0x578b100

Breakpoint 1, handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
988         gpi_to_user();

(gdb) print *cb_data
$1 = {reason = 1001, cb_rtn = 0x7f9136e84a61 <handle_vhpi_callback(vhpiCbDataT const*)>, obj = 0x500080 <deposit_doit+6954>, time = 0x5188808, value = 0x0,
  user_data = 0x57c5028}

(gdb) cont
Continuing.

Breakpoint 1, handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
988         gpi_to_user();

(gdb) print *cb_data
$2 = {reason = 1001, cb_rtn = 0x7f9136e84a61 <handle_vhpi_callback(vhpiCbDataT const*)>, obj = 0x500080 <deposit_doit+6954>, time = 0x5188808, value = 0x0,
  user_data = 0x57c5028}

(gdb) next
990         VhpiCbHdl *cb_hdl = (VhpiCbHdl *)cb_data->user_data;

(gdb) next
992         if (!cb_hdl) {

(gdb) next
998         gpi_cb_state_e old_state = cb_hdl->get_call_state();

(gdb) next
1000        if (old_state == GPI_PRIMED) {

(gdb) next
1001            cb_hdl->set_call_state(GPI_CALL);

(gdb) next
1002            cb_hdl->run_callback();

(gdb) step
GpiValueCbHdl::run_callback (this=0x57c5068) at cocotb/share/lib/gpi/GpiCbHdl.cpp:111
111         std::string current_value;


(gdb) next
114         if (required_value == "X")

(gdb) next
117             current_value = m_signal->get_signal_value_binstr();

(gdb) next
118             if (current_value == required_value) pass = true;

(gdb) next
118             if (current_value == required_value) pass = true;

(gdb) next
121         if (pass) {

(gdb) next
122             this->gpi_function(m_cb_data);

(gdb) step
handle_gpi_callback (user_data=0x57c1060) at cocotb/share/lib/simulator/simulatormodule.cpp:204
204         to_python();

(gdb) step
gpi_log (name=0x7f9135b12dcf "gpi", level=40, pathname=0x7f9135b12da0 "cocotb/share/lib/simulator/simulatormodule.cpp", funcname=0x7f9135b14910 <_ZZ19handle_gpi_callbackPvE8__func__.3170.3180> "handle_gpi_callback", lineno=204, msg=0x7f9135b12d80 "FATAL: We are calling up again") at cocotb/share/lib/gpi_log/gpi_logging.cpp:45
45          va_start(argp, msg);
46          gpi_vlog(name, level, pathname, funcname, lineno, msg, argp);
48      }

(gdb) next
[Inferior 1 (process 22060) exited with code 01]
```
Ok, so both times it is a `vhpiCbValueChange` callback (`cb_data->reason = 1001`).

Both times are also for the same `vhpiHandleT` (`cb_data->obj = 0x500080`).

The part that is important is what happens between the two breakpoints. Are you able to get a stack trace when the breakpoint is hit the second time? For this error I would expect the first callback to be on the stack still.
Sure, here is the backtrace at each time the breakpoint is hit:

```
+b handle_vhpi_callback
Breakpoint 1 at 0x7f1e2d38fa6d: file cocotb/share/lib/vhpi/VhpiImpl.cpp, line 988.
+handle SIGSEGV nostop noprint pass
Signal        Stop      Print   Pass to program Description
SIGSEGV       No        No      Yes             Segmentation fault
+cont
Continuing.

Breakpoint 1, handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
988         gpi_to_user();
+bt
#0  handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
#1  0x000000000082f364 in ?? ()
#2  0x0000000000c2a5a9 in ?? ()
#3  0x0000000000b6c4d8 in ?? ()
#4  0x0000000000b58d88 in ?? ()
#5  0x000000000089a84f in ?? ()
#6  0x00000000008992ca in vhpi_put_value ()
#7  0x00007f1e2d388710 in VhpiLogicSignalObjHdl::set_signal_value (this=0x5d68cc0, value=0, action=GPI_DEPOSIT) at cocotb/share/lib/vhpi/VhpiCbHdl.cpp:527
#8  0x00007f1e33164ad9 in gpi_set_signal_value_int (sig_hdl=0x5d68cc0, value=0, action=GPI_DEPOSIT) at cocotb/share/lib/gpi/GpiCommon.cpp:483
#9  0x00007f1e2c01cad6 in set_signal_val_int (self=0x7f1e2b0a2d70, args=0x7f1e2bff7180) at cocotb/share/lib/simulator/simulatormodule.cpp:657
#10 0x00007f1e2d081395 in cfunction_call_varargs (kwargs=<optimized out>, args=<optimized out>, func=0x7f1e2c0037c0) at Objects/call.c:757
#11 PyCFunction_Call (func=0x7f1e2c0037c0, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:772
#12 0x00007f1e2d057bb0 in do_call_core (kwdict=0x0, callargs=0x7f1e2bff7180, func=0x7f1e2c0037c0, tstate=0x5770c80) at Python/ceval.c:5007
#13 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3559
#14 0x00007f1e2d093559 in gen_send_ex (closing=0, exc=0, arg=<optimized out>, gen=0x7f1e2bfe6ac0) at Objects/genobject.c:222
#15 _PyGen_Send (gen=0x7f1e2bfe6ac0, arg=<optimized out>) at Objects/genobject.c:292
#16 0x00007f1e2d0896f5 in method_vectorcall_O (func=0x7f1e2c784ae0, args=0x7f1e2b588880, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:416
#17 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#18 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#19 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#20 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#21 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#22 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#23 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#24 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#25 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#26 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#27 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#28 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#29 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#30 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#31 0x00007f1e2d1726ee in _PyEval_EvalCodeWithName (_co=0x7f1e2b7112f0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=2,
    kwnames=0x7f1e2b773568, kwargs=kwargs@entry=0x7f1e2b586400, kwcount=1, kwstep=kwstep@entry=1, defs=defs@entry=0x7f1e2b767778, defcount=defcount@entry=1,
    kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name=0x7f1e2b768130, qualname=0x7f1e2b70f670) at Python/ceval.c:4298
#32 0x00007f1e2d07ecb2 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:435
#33 0x00007f1e2d082b31 in _PyObject_Vectorcall (kwnames=0x7f1e2b773550, nargsf=2, args=0x7f1e2b5863f0, callable=0x7f1e2b715d30) at ./Include/cpython/abstract.h:127
#34 method_vectorcall (method=<optimized out>, args=0x7f1e2b5863f8, nargsf=<optimized out>, kwnames=0x7f1e2b773550) at Objects/classobject.c:60
#35 0x00007f1e2d05661b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#36 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#37 call_function (kwnames=0x7f1e2b773550, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#38 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3515
#39 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#40 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#41 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#42 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#43 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#44 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#45 0x00007f1e2d082a54 in _PyObject_Vectorcall (kwnames=0x0, nargsf=2, args=0x7f1e36019950, callable=0x7f1e2b759dc0) at ./Include/cpython/abstract.h:127
#46 method_vectorcall (method=method@entry=0x7f1e2c4a4d00, args=0x7f1e2c245298, nargsf=nargsf@entry=1, kwnames=0x0) at Objects/classobject.c:89
#47 0x00007f1e2d080f0c in PyVectorcall_Call (callable=0x7f1e2c4a4d00, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:199
#48 0x00007f1e2c01bd8b in handle_gpi_callback (user_data=0x5d262a0) at cocotb/share/lib/simulator/simulatormodule.cpp:228
#49 0x00007f1e33166bff in GpiCommonCbHdl::run_callback (this=0x576ab38) at cocotb/share/lib/gpi/GpiCbHdl.cpp:86
#50 0x00007f1e333884c4 in handle_vpi_callback (cb_data=0x7f1e36019b10) at cocotb/share/lib/vpi/VpiImpl.cpp:652
#51 0x0000000000706c2e in ?? ()
#52 0x0000000000adee82 in ?? ()
#53 0x0000000004c2503a in ?? ()
#54 0x00007f1e36019bf8 in ?? ()
#55 0x0000000000000002 in ?? ()
#56 0x0000000000000000 in ?? ()
+cont
Continuing.

Breakpoint 1, handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
988         gpi_to_user();
+bt
#0  handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
#1  0x000000000082f364 in ?? ()
#2  0x0000000000c2a5a9 in ?? ()
#3  0x0000000000b6c4d8 in ?? ()
#4  0x0000000000b58d88 in ?? ()
#5  0x000000000089a84f in ?? ()
#6  0x00000000008992ca in vhpi_put_value ()
#7  0x00007f1e2d388710 in VhpiLogicSignalObjHdl::set_signal_value (this=0x5d68cc0, value=1, action=GPI_DEPOSIT) at cocotb/share/lib/vhpi/VhpiCbHdl.cpp:527
#8  0x00007f1e33164ad9 in gpi_set_signal_value_int (sig_hdl=0x5d68cc0, value=1, action=GPI_DEPOSIT) at cocotb/share/lib/gpi/GpiCommon.cpp:483
#9  0x00007f1e2c01cad6 in set_signal_val_int (self=0x7f1e2b0a2d70, args=0x7f1e2a923e00) at cocotb/share/lib/simulator/simulatormodule.cpp:657
#10 0x00007f1e2d081395 in cfunction_call_varargs (kwargs=<optimized out>, args=<optimized out>, func=0x7f1e2c007a40) at Objects/call.c:757
#11 PyCFunction_Call (func=0x7f1e2c007a40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:772
#12 0x00007f1e2d057bb0 in do_call_core (kwdict=0x0, callargs=0x7f1e2a923e00, func=0x7f1e2c007a40, tstate=0x5770c80) at Python/ceval.c:5007
#13 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3559
#14 0x00007f1e2d093559 in gen_send_ex (closing=0, exc=0, arg=<optimized out>, gen=0x7f1e2bfe6ac0) at Objects/genobject.c:222
#15 _PyGen_Send (gen=0x7f1e2bfe6ac0, arg=<optimized out>) at Objects/genobject.c:292
#16 0x00007f1e2d0896f5 in method_vectorcall_O (func=0x7f1e2c784ae0, args=0x7f1e2b588880, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:416
#17 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#18 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#19 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#20 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#21 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#22 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#23 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#24 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#25 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#26 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#27 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#28 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#29 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#30 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#31 0x00007f1e2d1726ee in _PyEval_EvalCodeWithName (_co=0x7f1e2b7112f0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=2,
    kwnames=0x7f1e2b773568, kwargs=kwargs@entry=0x7f1e2b586400, kwcount=1, kwstep=kwstep@entry=1, defs=defs@entry=0x7f1e2b767778, defcount=defcount@entry=1,
    kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0, name=0x7f1e2b768130, qualname=0x7f1e2b70f670) at Python/ceval.c:4298
#32 0x00007f1e2d07ecb2 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:435
#33 0x00007f1e2d082b31 in _PyObject_Vectorcall (kwnames=0x7f1e2b773550, nargsf=2, args=0x7f1e2b5863f0, callable=0x7f1e2b715d30) at ./Include/cpython/abstract.h:127
#34 method_vectorcall (method=<optimized out>, args=0x7f1e2b5863f8, nargsf=<optimized out>, kwnames=0x7f1e2b773550) at Objects/classobject.c:60
#35 0x00007f1e2d05661b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#36 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#37 call_function (kwnames=0x7f1e2b773550, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#38 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3515
#39 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#40 0x00007f1e2d050c4b in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#41 trace_call_function (kwnames=<optimized out>, nargs=<optimized out>, args=<optimized out>, func=<optimized out>, tstate=<optimized out>) at Python/ceval.c:4968
#42 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x5770c80) at Python/ceval.c:4984
#43 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3486
#44 0x00007f1e2d04eca0 in function_code_fastcall (co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:283
#45 0x00007f1e2d082a54 in _PyObject_Vectorcall (kwnames=0x0, nargsf=2, args=0x7f1e36019950, callable=0x7f1e2b759dc0) at ./Include/cpython/abstract.h:127
#46 method_vectorcall (method=method@entry=0x7f1e2c4a4d40, args=0x7f1e2a862058, nargsf=nargsf@entry=1, kwnames=0x0) at Objects/classobject.c:89
#47 0x00007f1e2d080f0c in PyVectorcall_Call (callable=0x7f1e2c4a4d40, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:199
#48 0x00007f1e2c01bd8b in handle_gpi_callback (user_data=0x5d7bc00) at cocotb/share/lib/simulator/simulatormodule.cpp:228
#49 0x00007f1e33166bff in GpiCommonCbHdl::run_callback (this=0x576ab38) at cocotb/share/lib/gpi/GpiCbHdl.cpp:86
#50 0x00007f1e333884c4 in handle_vpi_callback (cb_data=0x7f1e36019b10) at cocotb/share/lib/vpi/VpiImpl.cpp:652
#51 0x0000000000706c2e in ?? ()
#52 0x0000000000adee82 in ?? ()
#53 0x0000000004c2503a in ?? ()
#54 0x00007f1e36019bf8 in ?? ()
#55 0x0000000005bb1801 in ?? ()
#56 0xffffffffffffffff in ?? ()
#57 0x00000000057a0450 in ?? ()
#58 0x0000000000000000 in ?? ()
+cont
Continuing.
[Inferior 1 (process 31026) exited with code 01]
+quit
```
The main thing I spot is, that it is the same signal that is set in each call to gpi_set_signal_value_int. The first time it is set to 0, whereas the next time it is set to 1.
As for the following:
```
#1  0x000000000082f364 in ?? ()
#2  0x0000000000c2a5a9 in ?? ()
#3  0x0000000000b6c4d8 in ?? ()
#4  0x0000000000b58d88 in ?? ()
#5  0x000000000089a84f in ?? ()
```

it looks like these are functions in the simulator binary, `xmsim`, since these addresses belong to a memory mapped region of
the binary.
Yeah, it's what I expected.

These are the key parts:
```
#0  handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
[ ... simulator ... ]
#6  0x00000000008992ca in vhpi_put_value ()
#7  0x00007f1e2d388710 in VhpiLogicSignalObjHdl::set_signal_value (this=0x5d68cc0, value=1, action=GPI_DEPOSIT) at cocotb/share/lib/vhpi/VhpiCbHdl.cpp:527
#8  0x00007f1e33164ad9 in gpi_set_signal_value_int (sig_hdl=0x5d68cc0, value=1, action=GPI_DEPOSIT) at cocotb/share/lib/gpi/GpiCommon.cpp:483
#9  0x00007f1e2c01cad6 in set_signal_val_int (self=0x7f1e2b0a2d70, args=0x7f1e2a923e00) at cocotb/share/lib/simulator/simulatormodule.cpp:657
[ ... Python (cocotb) ... ]
#48 0x00007f1e2c01bd8b in handle_gpi_callback (user_data=0x5d7bc00) at cocotb/share/lib/simulator/simulatormodule.cpp:228
#49 0x00007f1e33166bff in GpiCommonCbHdl::run_callback (this=0x576ab38) at cocotb/share/lib/gpi/GpiCbHdl.cpp:86
#50 0x00007f1e333884c4 in handle_vpi_callback (cb_data=0x7f1e36019b10) at cocotb/share/lib/vpi/VpiImpl.cpp:652
```

1. While cocotb is running in response to a callback, it calls down through the `simulator` module to set a value, which goes to the GPI and then back to the simulator with `vhpi_put_value()`.
2. The simulator updates the value, checks for registered callbacks, then calls the callback immediately.

This is non-compliant with the VHDL LRM. The details are here: https://github.com/cocotb/cocotb/issues/2178#issuecomment-722789720
Okay, but why does it work the first time then? Also note, that frame 50 is handle_vpi_callback, not handle_vhpi_callback. And this is the case both times the breakpoint is hit.
You are right, I just noticed **VPI** then **VHPI**. Mixed language simulation might be part of the problem here.

Either way, we would expect the new callback to occur after the original callback returns.
Yes, I do have some Verilog modules deep down in the hierachy, but not any that I interact with from my testbench. Anyway, thank you for the debug effort.
If you step through the first breakpoint that "works", does it actually call up to Python, or is perhaps the trigger not primed?
From what I can deduce, it look like the callback is not really called, just cleaned up immediately.

```
+b handle_vhpi_callback
Breakpoint 1 at 0x7f0b6fe3aa6d: file cocotb/share/lib/vhpi/VhpiImpl.cpp, line 988.
+cont
Continuing.
warning: Corrupted shared library list: 0x65b1370 != 0x659d710

Breakpoint 1, handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:988
988         gpi_to_user();
+next
990         VhpiCbHdl *cb_hdl = (VhpiCbHdl *)cb_data->user_data;
+next
992         if (!cb_hdl) {
+next
998         gpi_cb_state_e old_state = cb_hdl->get_call_state();
+next
1000        if (old_state == GPI_PRIMED) {
+next
1001            cb_hdl->set_call_state(GPI_CALL);
+next
1002            cb_hdl->run_callback();
+step
GpiValueCbHdl::run_callback (this=0x65bb388) at cocotb/share/lib/gpi/GpiCbHdl.cpp:111
111         std::string current_value;
+next
114         if (required_value == "X")
+next
117             current_value = m_signal->get_signal_value_binstr();
+next
118             if (current_value == required_value) pass = true;
+next
118             if (current_value == required_value) pass = true;
+next
112         bool pass = false;
+next
121         if (pass) {
+next
124             cleanup_callback();
+next
125             arm_callback();
+next
128         return 0;
+next
129     }
+next
handle_vhpi_callback (cb_data=0x28d6ae0) at cocotb/share/lib/vhpi/VhpiImpl.cpp:1004
1004            gpi_cb_state_e new_state = cb_hdl->get_call_state();
+next
1007            if (new_state != GPI_PRIMED)
+next
1013        gpi_to_simulator();
+next
1014    };
+next
0x000000000082f364 in ?? ()
+quit
```
It looks like you have either a `RisingEdge` or `FallingEdge` Trigger being `await`-ed by a cocotb Task, but for this callback, the value doesn't match so it never propogates up to cocotb.

The next time, the value has changed to match the Trigger, so it does make it up to cocotb.
Okay, I see. Yes, I do have a RisingEdge trigger, and it should fire around 50ns into the simulation, which is the case the second time the callback is called, so that makes sense. The first time the callback is called, the simulation time is 0ns. Maybe that is when the signal in question is reset to zero.

Resolving patch:
diff --git a/.github/generate-envs.py b/.github/generate-envs.py
--- a/.github/generate-envs.py
+++ b/.github/generate-envs.py
@@ -255,16 +255,15 @@
         "python-version": "3.8",
         "group": "ci",
     },
-    # Xcelium VHDL (VHPI) is not yet supported.
-    # {
-    #     "lang": "vhdl",
-    #     "sim": "xcelium",
-    #     "sim-version": "cadence/xcelium/2309",
-    #     "os": "ubuntu-20.04",
-    #     "self-hosted": True,
-    #     "python-version": "3.8",
-    #     "group": "ci",
-    # },
+    {
+        "lang": "vhdl",
+        "sim": "xcelium",
+        "sim-version": "cadence/xcelium/2309",
+        "os": "ubuntu-20.04",
+        "self-hosted": True,
+        "python-version": "3.8",
+        "group": "ci",
+    },
 ]
 
 
diff --git a/src/cocotb/runner.py b/src/cocotb/runner.py
--- a/src/cocotb/runner.py
+++ b/src/cocotb/runner.py
@@ -1005,6 +1005,8 @@ def _get_parameter_options(parameters: Mapping[str, object]) -> Command:
     def _build_command(self) -> List[Command]:
         self.env["CDS_AUTO_64BIT"] = "all"
 
+        assert self.hdl_toplevel, "A HDL toplevel is required in all Xcelium compiles."
+
         verbosity_opts = []
         if self.verbose:
             verbosity_opts += ["-messages"]
@@ -1020,6 +1022,11 @@ def _build_command(self) -> List[Command]:
             verbosity_opts += ["-quiet"]
             verbosity_opts += ["-plinowarn"]
 
+        vhpi_opts = []
+        # Xcelium 23.09.004 fixes cocotb issue #1076 as long as the
+        # following define is set.
+        vhpi_opts.append("-NEW_VHPI_PROPAGATE_DELAY")
+
         cmds = [
             ["xrun"]
             + ["-logfile"]
@@ -1038,6 +1045,7 @@ def _build_command(self) -> List[Command]:
                 cocotb.config.lib_name_path("vpi", "xcelium")
                 + ":vlog_startup_routines_bootstrap"
             ]
+            + vhpi_opts
             + [f"-work {self.hdl_library}"]
             + self.build_args
             + ["-define COCOTB_SIM"]
@@ -1088,6 +1096,11 @@ def _test_command(self) -> List[Command]:
         else:
             input_tcl = ["-input", "@run; exit;"]
 
+        vhpi_opts = []
+        # Xcelium 23.09.004 fixes cocotb issue #1076 as long as the
+        # following define is set.
+        vhpi_opts.append("-NEW_VHPI_PROPAGATE_DELAY")
+
         cmds = [["mkdir", "-p", tmpdir]]
         cmds += [
             ["xrun"]
@@ -1098,6 +1111,7 @@ def _test_command(self) -> List[Command]:
             + ["-cds_implicit_tmpdir"]
             + [tmpdir]
             + ["-licqueue"]
+            + vhpi_opts
             + verbosity_opts
             + ["-R"]
             + self.test_args


Test patch:
diff --git a/examples/adder/tests/test_adder.py b/examples/adder/tests/test_adder.py
--- a/examples/adder/tests/test_adder.py
+++ b/examples/adder/tests/test_adder.py
@@ -69,6 +69,10 @@ def test_adder_runner():
     else:
         vhdl_sources = [proj_path / "hdl" / "adder.vhdl"]
 
+    build_test_args = []
+    if hdl_toplevel_lang == "vhdl" and sim == "xcelium":
+        build_test_args = ["-v93"]
+
     # equivalent to setting the PYTHONPATH environment variable
     sys.path.append(str(proj_path / "tests"))
 
@@ -78,8 +82,11 @@ def test_adder_runner():
         vhdl_sources=vhdl_sources,
         hdl_toplevel="adder",
         always=True,
+        build_args=build_test_args,
+    )
+    runner.test(
+        hdl_toplevel="adder", test_module="test_adder", test_args=build_test_args
     )
-    runner.test(hdl_toplevel="adder", test_module="test_adder")
 
 
 if __name__ == "__main__":
diff --git a/examples/matrix_multiplier/tests/test_matrix_multiplier.py b/examples/matrix_multiplier/tests/test_matrix_multiplier.py
--- a/examples/matrix_multiplier/tests/test_matrix_multiplier.py
+++ b/examples/matrix_multiplier/tests/test_matrix_multiplier.py
@@ -253,6 +253,8 @@ def test_matrix_multiplier_runner():
     extra_args = []
     if sim == "ghdl":
         extra_args = ["--std=08"]
+    elif sim == "xcelium":
+        extra_args = ["-v200x"]
 
     parameters = {
         "DATA_WIDTH": "32",
diff --git a/tests/pytest/test_plusargs.py b/tests/pytest/test_plusargs.py
--- a/tests/pytest/test_plusargs.py
+++ b/tests/pytest/test_plusargs.py
@@ -27,9 +27,9 @@ def test_toplevel_library():
 
     runner = get_runner(sim)
 
-    test_args = []
-    if sim == "xcelium":
-        test_args = ["-v93"]
+    build_test_args = []
+    if hdl_toplevel_lang == "vhdl" and sim == "xcelium":
+        build_test_args = ["-v93"]
 
     verilog_sources = []
     vhdl_sources = []
@@ -46,12 +46,13 @@ def test_toplevel_library():
         verilog_sources=verilog_sources,
         hdl_toplevel="tb_top",
         build_dir=str(test_module_path / "sim_build" / "pytest"),
+        build_args=build_test_args,
     )
 
     runner.test(
         hdl_toplevel="tb_top",
         test_module="plusargs",
-        test_args=test_args,
+        test_args=build_test_args,
         gpi_interfaces=gpi_interfaces,
         plusargs=["+foo=bar", "+test1", "+test2", "+options=fubar", "+lol=wow=4"],
     )
diff --git a/tests/pytest/test_vhdl_libraries_multiple.py b/tests/pytest/test_vhdl_libraries_multiple.py
--- a/tests/pytest/test_vhdl_libraries_multiple.py
+++ b/tests/pytest/test_vhdl_libraries_multiple.py
@@ -24,8 +24,8 @@
     reason="Skipping test since only VHDL is supported",
 )
 @pytest.mark.skipif(
-    os.getenv("SIM", "ghdl") not in ["ghdl", "questa", "riviera", "xcelium"],
-    reason="Skipping test since only GHDL, Questa/ModelSim, Riviera and Xcelium are supported",
+    os.getenv("SIM", "ghdl") not in ["ghdl", "questa", "riviera"],
+    reason="Skipping test since only GHDL, Questa/ModelSim, and Riviera are supported",
 )
 def test_toplevel_library():
     vhdl_gpi_interfaces = os.getenv("VHDL_GPI_INTERFACE", None)
diff --git a/tests/test_cases/issue_120/issue_120.py b/tests/test_cases/issue_120/issue_120.py
--- a/tests/test_cases/issue_120/issue_120.py
+++ b/tests/test_cases/issue_120/issue_120.py
@@ -20,12 +20,7 @@ async def monitor(dut):
     ), "stream_in_valid should be high on the 5th cycle"
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def issue_120_scheduling(dut):
     cocotb.start_soon(Clock(dut.clk, 10, "ns").start())
     cocotb.start_soon(monitor(dut))
diff --git a/tests/test_cases/issue_142/issue_142.py b/tests/test_cases/issue_142/issue_142.py
--- a/tests/test_cases/issue_142/issue_142.py
+++ b/tests/test_cases/issue_142/issue_142.py
@@ -6,12 +6,7 @@
 from cocotb.triggers import RisingEdge
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def issue_142_overflow_error(dut):
     """Tranparently convert ints too long to pass
     through the GPI interface natively into BinaryValues"""
diff --git a/tests/test_cases/issue_2255/Makefile b/tests/test_cases/issue_2255/Makefile
--- a/tests/test_cases/issue_2255/Makefile
+++ b/tests/test_cases/issue_2255/Makefile
@@ -14,6 +14,10 @@ endif
 
 TOPLEVEL := test
 
+ifneq ($(filter $(SIM),ius xcelium),)
+COMPILE_ARGS += -v93
+endif
+
 export MODULE := test_issue2255
 export COCOTB_LOG_LEVEL := DEBUG
 
diff --git a/tests/test_cases/issue_348/issue_348.py b/tests/test_cases/issue_348/issue_348.py
--- a/tests/test_cases/issue_348/issue_348.py
+++ b/tests/test_cases/issue_348/issue_348.py
@@ -48,34 +48,19 @@ async def start(self):
             assert mon, "Monitor saw nothing"
 
 
-# Cadence simulators: "Unable set up RisingEdge(ModifiableObject(sample_module.clk)) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def issue_348_rising(dut):
     """Start two monitors on RisingEdge"""
     await DualMonitor(RisingEdge, dut.clk).start()
 
 
-# Cadence simulators: "Unable set up FallingEdge(ModifiableObject(sample_module.clk)) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def issue_348_falling(dut):
     """Start two monitors on FallingEdge"""
     await DualMonitor(FallingEdge, dut.clk).start()
 
 
-# Cadence simulators: "Unable set up Edge(ModifiableObject(sample_module.clk)) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def issue_348_either(dut):
     """Start two monitors on Edge"""
     await DualMonitor(Edge, dut.clk).start()
diff --git a/tests/test_cases/test_cocotb/Makefile b/tests/test_cases/test_cocotb/Makefile
--- a/tests/test_cases/test_cocotb/Makefile
+++ b/tests/test_cases/test_cocotb/Makefile
@@ -27,6 +27,5 @@ MODULE := "\
 	test_start_soon,\
 	"
 
-ifeq ($(SIM),questa)
-	SIM_ARGS := "-t ps"
-endif
+# test_timing_triggers.py requires a 1ps time precision.
+COCOTB_HDL_TIMEPRECISION := 1ps
diff --git a/tests/test_cases/test_cocotb/test_clock.py b/tests/test_cases/test_cocotb/test_clock.py
--- a/tests/test_cases/test_cocotb/test_clock.py
+++ b/tests/test_cases/test_cocotb/test_clock.py
@@ -11,6 +11,7 @@
 import cocotb
 import pytest
 from cocotb.clock import Clock
+from cocotb.simulator import get_precision
 from cocotb.triggers import RisingEdge, Timer
 from cocotb.utils import get_sim_time
 
@@ -63,8 +64,14 @@ async def test_clock_with_units(dut):
     clk_gen.kill()
 
 
-@cocotb.test()
+# Xcelium/VHDL does not correctly report the simulator precision.
+# See also https://github.com/cocotb/cocotb/issues/3419
+@cocotb.test(skip=(cocotb.LANGUAGE == "vhdl" and cocotb.SIM_NAME.startswith("xmsim")))
 async def test_clocks_with_other_number_types(dut):
+    # The following test assumes a time precision of at least 0.1ns.
+    # Update the simulator invocation if this assert hits!
+    assert get_precision() <= -10
+
     clk1 = cocotb.start_soon(Clock(dut.clk, decimal.Decimal("1"), units="ns").start())
     await Timer(10, "ns")
     with pytest.warns(FutureWarning, match="cause a CancelledError to be thrown"):
diff --git a/tests/test_cases/test_cocotb/test_timing_triggers.py b/tests/test_cases/test_cocotb/test_timing_triggers.py
--- a/tests/test_cases/test_cocotb/test_timing_triggers.py
+++ b/tests/test_cases/test_cocotb/test_timing_triggers.py
@@ -17,6 +17,7 @@
 import cocotb
 import pytest
 from cocotb.clock import Clock
+from cocotb.simulator import get_precision
 from cocotb.triggers import (
     First,
     Join,
@@ -42,8 +43,20 @@ async def test_function_reentrant_clock(dut):
         await timer
 
 
-@cocotb.test()
+# Xcelium/VHDL does not correctly report the simulator precision.
+# See also https://github.com/cocotb/cocotb/issues/3419
+# NVC and GHDL do not support setting precision and always use 1 fs.
+@cocotb.test(
+    skip=(
+        cocotb.LANGUAGE == "vhdl"
+        and cocotb.SIM_NAME.lower().startswith(("xmsim", "ghdl", "nvc"))
+    )
+)
 async def test_timer_with_units(dut):
+    # The following test assumes a time precision of 1ps. Update the simulator
+    # invocation if this assert hits!
+    assert get_precision() == -12
+
     time_fs = get_sim_time(units="fs")
 
     # Await for one simulator time step
diff --git a/tests/test_cases/test_external/test_external.py b/tests/test_cases/test_external/test_external.py
--- a/tests/test_cases/test_external/test_external.py
+++ b/tests/test_cases/test_external/test_external.py
@@ -86,12 +86,7 @@ async def test_time_in_external(dut):
     assert time == time_now
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def test_time_in_function(dut):
     """
     Test that an @external function calling back into a cocotb @function
@@ -119,12 +114,7 @@ def wait_cycles_wrapper(dut, n):
             assert expected_after == time_after
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def test_external_call_return(dut):
     """
     Test ability to await an external function that is not a coroutine using @external
@@ -183,12 +173,7 @@ async def test_function_from_readonly(dut):
     assert value == 2
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def test_function_that_awaits(dut):
     """
     Test that @external functions can call @function coroutines that
@@ -201,12 +186,7 @@ async def test_function_that_awaits(dut):
     assert value == 2
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def test_await_after_function(dut):
     """
     Test that awaiting a Trigger works after returning
@@ -222,12 +202,7 @@ async def test_await_after_function(dut):
     await RisingEdge(dut.clk)
 
 
-# Cadence simulators: "Unable set up RisingEdge(...) Trigger" with VHDL (see #1076)
-@cocotb.test(
-    expect_error=cocotb.triggers._TriggerException
-    if cocotb.SIM_NAME.startswith(("xmsim", "ncsim")) and cocotb.LANGUAGE in ["vhdl"]
-    else ()
-)
+@cocotb.test()
 async def test_external_from_start_soon(dut):
     """
     Test that @external functions work when awaited from a forked
diff --git a/tests/test_cases/test_iteration_mixedlang/test_iteration.py b/tests/test_cases/test_iteration_mixedlang/test_iteration.py
--- a/tests/test_cases/test_iteration_mixedlang/test_iteration.py
+++ b/tests/test_cases/test_iteration_mixedlang/test_iteration.py
@@ -73,10 +73,6 @@ async def test_loads(dut):
         tlog.info("Found %s" % repr(load))
 
 
-# Xcelium fails with "FATAL: We are calling up again".
-# TODO: Re-evaluate waiver once https://github.com/cocotb/cocotb/issues/1076 is
-# fixed.
-@cocotb.test(skip=cocotb.SIM_NAME.startswith("xmsim"))
 async def recursive_discovery(dut):
     """Recursively discover every single object in the design."""
     if cocotb.SIM_NAME.lower().startswith(("ncsim", "xmsim")):
@@ -101,10 +97,6 @@ async def recursive_discovery(dut):
     )
 
 
-# Xcelium fails with "FATAL: We are calling up again".
-# TODO: Re-evaluate waiver once https://github.com/cocotb/cocotb/issues/1076 is
-# fixed.
-@cocotb.test(skip=cocotb.SIM_NAME.startswith("xmsim"))
 async def recursive_discovery_boundary(dut):
     """Iteration through the boundary works but this just double checks."""
     if cocotb.SIM_NAME.lower().startswith(("ncsim", "xmsim")):
diff --git a/tests/test_cases/test_long_log_msg/Makefile b/tests/test_cases/test_long_log_msg/Makefile
--- a/tests/test_cases/test_long_log_msg/Makefile
+++ b/tests/test_cases/test_long_log_msg/Makefile
@@ -20,6 +20,10 @@ else ifeq ($(TOPLEVEL_LANG),vhdl)
 VHDL_SOURCES := $(PROJ_DIR)/test.vhd
 endif
 
+ifneq ($(filter $(SIM),ius xcelium),)
+COMPILE_ARGS += -v93
+endif
+
 TOPLEVEL := test
 
 export MODULE := test_long_log_msg