Repository: LLNL/Caliper (language: python)

Issue:
Region stack mismatch error
Hello,

I am trying to get a basic count of the MPI calls in an app that uses Kokkos.

Since the app uses Kokkos,  I set the `KOKKOS_TOOLS_LIBS` env variable to point to `libcaliper.so` and did not add any additional Caliper annotations to the source code.

To get the MPI counts, I set the `CALI_CONFIG` env variable to `mpi-report`. I have also tried setting `CALI_CONFIG="runtime-report(mpi.message.count)"`.

 When I run the app with either of these configs, I get these warnings (one per process):
 
 ```
 == CALIPER: (1): Region stack mismatch: Trying to end
    "mpi.call.id"
  but region stack is empty!
  Ceasing region tracking!
== CALIPER: (2): Region stack mismatch: Trying to end
    "mpi.call.id"
  but region stack is empty!
  Ceasing region tracking!
== CALIPER: (3): Region stack mismatch: Trying to end
    "mpi.call.id"
  but region stack is empty!
  Ceasing region tracking!
== CALIPER: (0): Region stack mismatch: Trying to end
    "mpi.call.id"
  but region stack is empty!
 ```

Caliper still outputs a report at the end of the run, but the MPI counts are not complete (I know that there should be more counts than are present). I suspect that this `Region stack mismatch` causes Caliper to not be able to profile in some regions.

In what contexts does this warning show up? And how can I resolve it?


Discussion / hints:
Hi @cwschilly, thanks for the report. I've seen this happen if an MPI call internally invokes other MPI API functions, which they're not really supposed to do. Which MPI library version are you using?

I can work around it but it'll require a fix in Caliper.
Thanks for the response @daboehme, I'm using OpenMPI (v4.1.5).
Hi @daboehme , I also get this error when I use runtime-report; any idea what might cause this?

```
cali-query: TreeFormatter: Attribute "min#sum#sum#time.duration" not found.
cali-query: TreeFormatter: Attribute "max#sum#sum#time.duration" not found.
cali-query: TreeFormatter: Attribute "avg#sum#sum#time.duration" not found.
cali-query: TreeFormatter: Attribute "percent_total#sum#sum#time.duration" not found.
```
Hm that's interesting. Is this with the `mpi.message.count` option? If so can you try it without? Otherwise what Caliper version are you using?
@daboehme No, it's with no options selected: `export CALI_CONFIG=runtime-report`.

I'm using Caliper v2.9.1. I tried with 2.10 but all the same warnings are present.

Fix patch:
diff --git a/src/services/mpiwrap/MpiWrap.cpp b/src/services/mpiwrap/MpiWrap.cpp
index 15563da8d..6d58b4141 100644
--- a/src/services/mpiwrap/MpiWrap.cpp
+++ b/src/services/mpiwrap/MpiWrap.cpp
@@ -20,7 +20,6 @@ Attribute mpifn_attr       { Attribute::invalid };
 Attribute mpirank_attr     { Attribute::invalid };
 Attribute mpisize_attr     { Attribute::invalid };
 Attribute mpicall_attr     { Attribute::invalid };
-Attribute mpi_call_id_attr { Attribute::invalid };
 
 extern void mpiwrap_init(Caliper* c, Channel* chn, cali::ConfigSet& cfg);
 
@@ -72,12 +71,6 @@ void mpi_register(Caliper* c, Channel* chn)
             c->create_attribute("mpi.world.size", CALI_TYPE_INT,
                                 CALI_ATTR_GLOBAL        |
                                 CALI_ATTR_SKIP_EVENTS);
-    if (mpi_call_id_attr == Attribute::invalid)
-        mpi_call_id_attr =
-            c->create_attribute("mpi.call.id", CALI_TYPE_UINT,
-                                CALI_ATTR_SCOPE_THREAD  |
-                                CALI_ATTR_ASVALUE       |
-                                CALI_ATTR_SKIP_EVENTS);
 
     ConfigSet cfg = services::init_config_from_spec(chn->config(), mpi_service_spec);
 
diff --git a/src/services/mpiwrap/Wrapper.w b/src/services/mpiwrap/Wrapper.w
index 706469022..04da53622 100644
--- a/src/services/mpiwrap/Wrapper.w
+++ b/src/services/mpiwrap/Wrapper.w
@@ -31,7 +31,7 @@ namespace cali
 extern Attribute mpifn_attr;
 extern Attribute mpirank_attr;
 extern Attribute mpisize_attr;
-extern Attribute mpi_call_id_attr;
+// extern Attribute mpi_call_id_attr;
 
 }
 
@@ -52,12 +52,12 @@ int  {{foo}}_wrap_count = 0;
 
 inline void push_mpifn(Caliper* c, bool enabled, const char* fname)
 {
-    static std::atomic<uint64_t> call_id { 0 };
+    // static std::atomic<uint64_t> call_id { 0 };
 
     if (!enabled)
         return;
 
-    c->begin(mpi_call_id_attr, Variant(cali_make_variant_from_uint(++call_id)));
+    // c->begin(mpi_call_id_attr, Variant(cali_make_variant_from_uint(++call_id)));
     c->begin(mpifn_attr, Variant(fname));
 }
 
@@ -67,7 +67,7 @@ inline void pop_mpifn(Caliper* c, bool enabled)
         return;
 
     c->end(mpifn_attr);
-    c->end(mpi_call_id_attr);
+    // c->end(mpi_call_id_attr);
 }
 
 


Test patch:
diff --git a/test/ci_app_tests/test_mpi.py b/test/ci_app_tests/test_mpi.py
index 58528b288..13550d1da 100644
--- a/test/ci_app_tests/test_mpi.py
+++ b/test/ci_app_tests/test_mpi.py
@@ -164,7 +164,7 @@ def test_mpi_msg_trace(self):
                          'mpi.comm.is_world' : 'true'
             }))
         self.assertTrue(cat.has_snapshot_with_keys(
-            snapshots, { 'region', 'mpi.function', 'mpi.coll.type', 'mpi.call.id'
+            snapshots, { 'region', 'mpi.function', 'mpi.coll.type'
             }))
 
     def test_mpireport_controller(self):