Repository: ocaml/ocaml
Pull request #14025: domain_terminate: orphan the shared heap before STW exit
Author: login: gasche
Labels: name: bug
name: runtime-system
Created: 2025-05-11T07:50:34Z | Merged: 2025-05-20T10:43:48Z

This is a proposal to try to fix a compaction bug discussed in #13739 (cc @sadiq, @jmid).

In that issue, @jmid is tracking a rare failure in the compaction code when run concurrently with other domain activity. He observed with @sadiqj that there is a race condition between the compactor code, that assumes that all pools are owned by an active domain, and the domain-termination code, which waits until after leaving the STW participant set to orphan its shared heap.

The current PR changes the domain-termination logic to orphan the shared heap before leaving the STW participant set, so that the compactor's assumptions are correct. This is tricky code (that part of the domain-termination logic runs in a loop while answering STW requests) which would need a review or rewrite by concurrent-runtime experts.

To make the present issue self-contained, let me copy below the more detailed explanations I wrote in #13739:

> - Context. The shared heap is a set of "pools". Each domain owns a subset of these pools -- they are in its "local" shared heaps. A domain will allocate/promote new block in its own "local" pools, and take care of those during GC events. There is also a global structure `pool_freelist` that contains "orphan" pools, which are not owned by any domain: when a domain terminates, it pushes its own pools in the free list. Unlike what the name might suggest, the pools in the freelist are not free, they may contain OCaml values. Domains may "adopt" pools from the freelist if they want more space and their local pools are all full.
> 
>  - `caml_cycle_heap` (which happens in a STW section at the end of each major GC cycle) will empty the global freelist by forcing one domain to adopt all its pools. I guess that this is intended to guarantee that all pools in the freelist are swept *eventually*, they don't remain in global limbo forever.
> 
>  - The GC compaction code needs a full view of the OCaml heap, because it will move values around and needs to update the whole heap to update pointers to moved values. This was done by calling the compactor after `caml_cycle_heap`, to guarantee that free list pools have been adopted, and so that all pools are in the local heap of some active domain (a participant to the STW set).
> 
> - In fact this guarantee does not hold, because terminating domains move their pools into the freelist _asynchronously_, after removing themselves from the STW participant set. (The general idea of termination code is to do as much of the book-keeping work as possible after leaving the STW set.)
> 
> - In particular, a terminating domain may go to sleep after removing itself from the STW set and before moving its heap into the major heap, and the rest of the runtime will proceed happily, including by running the compaction code, while some of the shared heap is "missing" (neither in an active domain nor in the global heap). This is not a problem for the normal runtime, the prodigious pools will be moved to the freelist on their return. But it is a bug for compaction, because if compaction happened those returning pools will be in an invalid state -- they have not been updated when values in other pools were moved around.





Diff:
diff --git a/Changes b/Changes
index de844bac42e9..d22c25ccf81b 100644
--- a/Changes
+++ b/Changes
@@ -755,6 +755,10 @@ OCaml 5.4.0
   demarshaling.
   (Benoît Vaugon, review by David Allsopp and Gabriel Scherer)
 
+- #14025: fix data race between compaction and domain termination
+  (Gabriel Scherer, review by Jan Midtgaard,
+   report by Jan Midtgaard)
+
 OCaml 5.3.0 (8 January 2025)
 ----------------------------
 
diff --git a/runtime/caml/shared_heap.h b/runtime/caml/shared_heap.h
index 57f4963ac521..898e9a533e71 100644
--- a/runtime/caml/shared_heap.h
+++ b/runtime/caml/shared_heap.h
@@ -31,7 +31,15 @@ struct caml_heap_state;
 struct pool;
 
 struct caml_heap_state* caml_init_shared_heap(void);
-void caml_teardown_shared_heap(struct caml_heap_state* heap);
+void caml_adopt_all_orphan_heaps(struct caml_heap_state* heap);
+void caml_assert_shared_heap_is_empty(struct caml_heap_state *heap);
+
+// ensures that the shared heap is empty
+void caml_orphan_shared_heap(struct caml_heap_state* heap);
+
+// requires that the shared heap is empty
+void caml_free_shared_heap(struct caml_heap_state* heap);
+
 
 value* caml_shared_try_alloc(struct caml_heap_state*,
                              mlsize_t, tag_t, reserved_t);
diff --git a/runtime/domain.c b/runtime/domain.c
index e899bb691a6d..c928d89a0766 100644
--- a/runtime/domain.c
+++ b/runtime/domain.c
@@ -778,7 +778,9 @@ static void domain_create(uintnat initial_minor_heap_wsize,
 reallocate_minor_heap_failure:
   caml_teardown_major_gc();
 init_major_gc_failure:
-  caml_teardown_shared_heap(d->state->shared_heap);
+  caml_orphan_shared_heap(d->state->shared_heap);
+  caml_free_shared_heap(d->state->shared_heap);
+  domain_state->shared_heap = NULL;
 init_shared_heap_failure:
   caml_free_minor_tables(domain_state->minor_tables);
   domain_state->minor_tables = NULL;
@@ -2042,6 +2044,12 @@ int caml_domain_is_terminating (void)
   return domain_terminating(domain_self);
 }
 
+static bool marking_and_sweeping_done(caml_domain_state *domain_state)
+{
+  return (domain_state->marking_done
+          && domain_state->sweeping_done);
+}
+
 void caml_domain_terminate(bool last)
 {
   caml_domain_state* domain_state = domain_self->state;
@@ -2072,14 +2080,28 @@ void caml_domain_terminate(bool last)
     caml_orphan_ephemerons(domain_state);
     caml_orphan_finalisers(domain_state);
 
+    /* Orphaning ephemerons and finalizers may create new marking or
+       sweeping work, so we may need to mark and/or sweep again. */
+
     /* No need to check for interrupts if we are the last domain running. */
     if (last) {
       CAML_EV_LIFECYCLE(EV_DOMAIN_TERMINATE, getpid());
       break;
     }
 
-    /* take the all_domains_lock to try and exit the STW participant set
-       without racing with a STW section being triggered */
+    /* If new marking or sweeping work appeared during orphaning,
+       run a new loop iteration. */
+    if (!marking_and_sweeping_done(domain_state))
+      continue;
+
+    /* Orphan the local shared heap.
+       This is only valid when [sweeping_done], and does
+       not create any new major GC work. */
+    caml_orphan_shared_heap(domain_state->shared_heap);
+    CAMLassert(marking_and_sweeping_done(domain_state));
+
+    /* Take the all_domains_lock to try and exit the STW participant set
+       without racing with a STW section being triggered. */
     caml_plat_lock_blocking(&all_domains_lock);
 
     /* The interaction of termination and major GC is quite subtle.
@@ -2089,23 +2111,19 @@ void caml_domain_terminate(bool last)
        require this domain to participate, which in turn could involve a major
        GC cycle. This would then require finish marking and sweeping again in
        order to decrement the globals [num_domains_to_mark] and
-       [num_domains_to_sweep] (see major_gc.c).
+       [num_domains_to_sweep] (see major_gc.c). We do this by running a new
+       loop iteration.
      */
-
-    if (!caml_incoming_interrupts_queued() &&
-        domain_state->marking_done &&
-        domain_state->sweeping_done) {
-
+    if (!caml_incoming_interrupts_queued()) {
       finished = 1;
       s->terminating = 0;
       s->running = 0;
 
-      /* Remove this domain from stw_domains */
+      /* Remove this domain from stw_domains. */
       remove_from_stw_domains(domain_self);
 
-      /* signal the interruptor condition variable
-       * because the backup thread may be waiting on it
-       */
+      /* Signal the interruptor condition variable
+         because the backup thread may be waiting on it. */
       caml_plat_lock_blocking(&s->lock);
       caml_plat_broadcast(&s->cond);
       caml_plat_unlock(&s->lock);
@@ -2118,19 +2136,22 @@ void caml_domain_terminate(bool last)
     caml_plat_unlock(&all_domains_lock);
   }
 
-  /* domain_state may be re-used by a fresh domain here (now that we
-   * have done remove_from_stw_domains and released the
-   * all_domains_lock). However, domain_create() won't touch it until
-   * it has claimed the domain_lock, so we hang onto that while we are
-   * tearing down the state. */
+  if (!last) caml_assert_shared_heap_is_empty(domain_state->shared_heap);
+
+  /* [domain_state] may be re-used by a fresh domain here, now that we
+     have done [remove_from_stw_domains] and released the
+     [all_domains_lock]. In particular, we cannot touch
+     [domain_self->interruptor] after here because it may be reused.
+
+     However, [domain_create()] won't touch the domain state until
+     it has claimed the [domain_lock], so we hang onto that while we are
+     tearing down the state. */
 
   /* Delete the domain state from statmemprof after any promotion
    * (etc) done by this domain: any remaining memprof state will be
    * handed over to surviving domains. */
   caml_memprof_delete_domain(domain_state);
 
-  /* We can not touch domain_self->interruptor after here
-     because it may be reused */
   caml_remove_generational_global_root(&domain_state->dls_root);
   caml_remove_generational_global_root(&domain_state->backtrace_last_exn);
   caml_stat_free(domain_state->final_info);
@@ -2139,21 +2160,33 @@ void caml_domain_terminate(bool last)
   caml_free_extern_state();
   caml_teardown_major_gc();
 
-  if (last)
+  /* At this point, we know that the shared heap has been orphaned,
+     except if [last], if we are the last domain. In that case we
+     finalise all unswept objects and orphan the shared heap now. */
+  if (last) {
+    /* First adopt all orphan pools, to avoid missing unswept objects. */
+    caml_adopt_all_orphan_heaps(domain_state->shared_heap);
+
+    /* Call all custom finalisers of unswept objects. */
     caml_finalise_heap();
 
-  caml_teardown_shared_heap(domain_state->shared_heap);
-  domain_state->shared_heap = 0;
-  caml_free_minor_tables(domain_state->minor_tables);
-  domain_state->minor_tables = 0;
+    /* Then orphan all pools again. */
+    caml_orphan_shared_heap(domain_state->shared_heap);
+  }
+  caml_assert_shared_heap_is_empty(domain_state->shared_heap);
 
-  caml_orphan_alloc_stats(domain_state);
-  /* Heap stats were orphaned by [caml_teardown_shared_heap] above.
-     At this point, the stats of the domain must be empty.
+  caml_free_shared_heap(domain_state->shared_heap);
+  domain_state->shared_heap = NULL;
+  caml_free_minor_tables(domain_state->minor_tables);
+  domain_state->minor_tables = NULL;
 
-     The sampled copy was also cleared by the minor collection(s)
-     performed above at [caml_empty_minor_heaps_once()], see the
-     termination-specific logic in [caml_collect_gc_stats_sample_stw].
+  /* At this point, the stats of the domain must be empty.
+     - heap stats were orphaned by [caml_orphan_shared_heap]
+     - alloc stats were orphaned by [caml_orphan_alloc_stats]
+     - the sampled copy in [sampled_gc_stats] was cleared by the minor
+       collection performed by [caml_empty_minor_heaps_once()], see
+       the termination-specific logic in
+       [caml_collect_gc_stats_sample_stw].
   */
 
   /* TODO: can this ever be NULL? can we remove this check? */
diff --git a/runtime/shared_heap.c b/runtime/shared_heap.c
index cae6235cbd15..b9b265c65acc 100644
--- a/runtime/shared_heap.c
+++ b/runtime/shared_heap.c
@@ -112,6 +112,8 @@ struct compact_pool_stat {
 static void orphan_heap_stats_with_lock(struct caml_heap_state *);
 static void adopt_pool_stats_with_lock(struct caml_heap_state *,
                                        pool *, sizeclass);
+static void adopt_all_pool_stats_with_lock(struct caml_heap_state *adopter);
+
 
 struct caml_heap_state* caml_init_shared_heap (void) {
   struct caml_heap_state* heap;
@@ -146,7 +148,7 @@ static int move_all_pools(pool** src, _Atomic(pool*)* dst,
   return count;
 }
 
-void caml_teardown_shared_heap(struct caml_heap_state* heap) {
+void caml_orphan_shared_heap(struct caml_heap_state* heap) {
   int released = 0, released_large = 0;
 
   caml_plat_lock_blocking(&pool_freelist.lock);
@@ -173,11 +175,57 @@ void caml_teardown_shared_heap(struct caml_heap_state* heap) {
   }
   orphan_heap_stats_with_lock(heap);
   caml_plat_unlock(&pool_freelist.lock);
-  caml_stat_free(heap);
-  caml_gc_log("Shutdown shared heap. Released %d active pools, %d large",
+  caml_gc_log("Orphan shared heap. Released %d active pools, %d large",
               released, released_large);
 }
 
+void caml_adopt_all_orphan_heaps(struct caml_heap_state* local) {
+  int received_p = 0, received_l = 0;
+  caml_plat_lock_blocking(&pool_freelist.lock);
+  for (int i = 0; i < NUM_SIZECLASSES; i++) {
+    received_p += move_all_pools(
+        (pool**)&pool_freelist.global_avail_pools[i],
+        (_Atomic(pool*)*)&local->unswept_avail_pools[i],
+        local->owner);
+    received_p += move_all_pools(
+        (pool**)&pool_freelist.global_full_pools[i],
+        (_Atomic(pool*)*)&local->unswept_full_pools[i],
+        local->owner);
+  }
+  while (pool_freelist.global_large) {
+    large_alloc* a = pool_freelist.global_large;
+    pool_freelist.global_large = a->next;
+    a->owner = local->owner;
+    a->next = local->unswept_large;
+    local->unswept_large = a;
+    received_l++;
+  }
+  if (received_p || received_l) {
+    adopt_all_pool_stats_with_lock(local);
+  }
+  caml_plat_unlock(&pool_freelist.lock);
+  if (received_p || received_l)
+    caml_gc_log("Received %d new pools, %d new large allocs",
+                received_p, received_l);
+  local->next_to_sweep = 0;
+}
+
+void caml_assert_shared_heap_is_empty(struct caml_heap_state* heap) {
+  for (int i = 0; i < NUM_SIZECLASSES; i++) {
+    CAMLassert(!heap->avail_pools[i]);
+    CAMLassert(!heap->full_pools[i]);
+    CAMLassert(!heap->unswept_avail_pools[i]);
+    CAMLassert(!heap->unswept_full_pools[i]);
+  }
+  CAMLassert(!heap->unswept_large);
+  CAMLassert(!heap->swept_large);
+}
+
+void caml_free_shared_heap(struct caml_heap_state* heap) {
+  caml_assert_shared_heap_is_empty(heap);
+  caml_stat_free(heap);
+}
+
 
 /* Allocating and deallocating pools from the global freelist. */
 
@@ -663,13 +711,13 @@ void caml_finalise_heap(void) {
   struct caml_heap_state *local = Caml_state->shared_heap;
   sizeclass sz;
 
-  /* local pools */
+  /* Finalise and release unswept local pools. */
   for (sz = 0; sz < NUM_SIZECLASSES; sz++) {
     pool_finalise(local, &local->unswept_avail_pools[sz], sz);
     pool_finalise(local, &local->unswept_full_pools[sz], sz);
   }
 
-  /* global pools */
+  /* Finalise and free large unswept objects. */
   if (local->unswept_large)
     large_alloc_finalise(local);
 }
@@ -1488,8 +1536,6 @@ void caml_cycle_heap_from_stw_single (void) {
 }
 
 void caml_cycle_heap(struct caml_heap_state* local) {
-  int received_p = 0, received_l = 0;
-
   caml_gc_log("Cycling heap [%02d]", local->owner->id);
   for (int i = 0; i < NUM_SIZECLASSES; i++) {
     CAMLassert(local->unswept_avail_pools[i] == NULL);
@@ -1503,34 +1549,7 @@ void caml_cycle_heap(struct caml_heap_state* local) {
   local->unswept_large = local->swept_large;
   local->swept_large = NULL;
 
-  caml_plat_lock_blocking(&pool_freelist.lock);
-  for (int i = 0; i < NUM_SIZECLASSES; i++) {
-    received_p += move_all_pools(
-        (pool**)&pool_freelist.global_avail_pools[i],
-        (_Atomic(pool*)*)&local->unswept_avail_pools[i],
-        local->owner);
-    received_p += move_all_pools(
-        (pool**)&pool_freelist.global_full_pools[i],
-        (_Atomic(pool*)*)&local->unswept_full_pools[i],
-        local->owner);
-  }
-  while (pool_freelist.global_large) {
-    large_alloc* a = pool_freelist.global_large;
-    pool_freelist.global_large = a->next;
-    a->owner = local->owner;
-    a->next = local->unswept_large;
-    local->unswept_large = a;
-    received_l++;
-  }
-  if (received_p || received_l) {
-    adopt_all_pool_stats_with_lock(local);
-  }
-  caml_plat_unlock(&pool_freelist.lock);
-  if (received_p || received_l)
-    caml_gc_log("Received %d new pools, %d new large allocs",
-                received_p, received_l);
-
-  local->next_to_sweep = 0;
+  caml_adopt_all_orphan_heaps(local);
 }
 
 void caml_finalise_freelist(void) {


Reviews:
user: login: gasche
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-11T07:53:27Z
user: login: sadiqj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T20:49:42Z
user: login: gasche
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-18T21:20:59Z
user: login: gasche
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-18T22:27:59Z
user: login: sadiqj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T10:54:17Z
user: login: jmid
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T14:49:39Z
user: login: gasche
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T15:06:06Z
user: login: gasche
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T15:15:24Z
user: login: sadiqj
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T16:35:52Z

Review comments:
diff_hunk: @@ -778,6 +778,7 @@ static void domain_create(uintnat initial_minor_heap_wsize,
 reallocate_minor_heap_failure:
   caml_teardown_major_gc();
 init_major_gc_failure:
+  caml_orphan_shared_heap(d->state->shared_heap);
path: runtime/domain.c
user: login: gasche
body: Minor: I suspect this call is actually a no-op because there should be no major-heap allocation during domain initialization (after heap initialization and before the domain starts running mutator code), but it does not hurt to have it for robustness (maybe later the domain-initialization code will start using the major heap?).
created_at: 2025-05-11T07:53:27Z
author_association: MEMBER
diff_hunk: @@ -2072,6 +2074,11 @@ void caml_domain_terminate(bool last)
     caml_orphan_ephemerons(domain_state);
     caml_orphan_finalisers(domain_state);
 
+    if (domain_state->marking_done &&
+        domain_state->sweeping_done) {
path: runtime/domain.c
user: login: sadiqj
body: I'm not sure I understand these conditions.

We've emptied our mark stack a few lines earlier (which means `marking_done` is 1). There's only a two places this can go back to 0, when we cycle the heap and in [`caml_darken`](https://github.com/gasche/ocaml/blob/14ec1c2e78566f7e25a533c7a18ca93567af96d2/runtime/major_gc.c#L1472-L1475). I can't see any paths to `caml_darken` that take a foreign `state` and so I don't think it can happen between when we empty the mark stack and this point.

On the other hand `sweeping_done` being 0 could be triggered by us having cycled the heap during the attempt to empty the minor heap?

Just stepping back, the only thing we really care about is whether the pools have been swept before we orphan them. Whether marking has happened or not is irrelevant, reachable blocks will be marked regardless.

Since `sweeping_done` can only be reset at the end of a cycle (unlike `marking_done`) I think we could simplify this PR by just orphaning the pools directly after `caml_finish_sweeping`.
created_at: 2025-05-18T20:49:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2072,6 +2074,11 @@ void caml_domain_terminate(bool last)
     caml_orphan_ephemerons(domain_state);
     caml_orphan_finalisers(domain_state);
 
+    if (domain_state->marking_done &&
+        domain_state->sweeping_done) {
path: runtime/domain.c
user: login: gasche
body: The conditions are sufficient because if they do not hold, then we do not enter the `finished = 1` block below, and we do another turn of the loop. We only need to do this orphaning on the last iteration of the loop (but it does not hurt to do it before).

(Now that I write this explanation, I'm thinking again about the `if (last)` case that exits earlier, possibly without orphaning the major heap. In this case there cannot be a race with another domain compacting, so we are fine regarding compactor races, but the CAMLassert invariants on `caml_free_shared_heap` are not respected. I could remove the assertions in `caml_free_shared_heap`, or call `caml_orphan_shared_heap` at the end of `caml_finalise_heap()` which is called in the `(last)` case after exiting the loop.)

Regarding whether the conditions are necessary: I am not sure (I'm not sure it really matters), but note that `caml_orphan_ephemerons` calls `ephe_mark` which calls `caml_darken` which can set `making_done = 0`, and also `caml_orphan_finalisers` can call `caml_finish_major_gc_cycle` which runs a STW section, at which point anything can happen.

I will try to refactor the code a bit to:
- Clarify that this condition is designed to be the same as the loop-exit condition to enter the `finished = 1` block below
- Document which invariants hold when we exit the loop, and whether they depend on `(last)`. I wonder if we should consider removing the fast-path exit (and instead doing another loop iteration) when sweeping and marking are not done in the `(last)` case, to simplify the reasoning about what is going on.
created_at: 2025-05-18T21:20:59Z
author_association: MEMBER
diff_hunk: @@ -2072,6 +2074,11 @@ void caml_domain_terminate(bool last)
     caml_orphan_ephemerons(domain_state);
     caml_orphan_finalisers(domain_state);
 
+    if (domain_state->marking_done &&
+        domain_state->sweeping_done) {
path: runtime/domain.c
user: login: gasche
body: I pushed a new commit that hopefully clarifies the invariants involved. This was more invasive than I would have hoped, and I wouldn't fully trust it without a new round of testing by @jmid if possible.
created_at: 2025-05-18T22:27:59Z
author_association: MEMBER
diff_hunk: @@ -2072,6 +2074,11 @@ void caml_domain_terminate(bool last)
     caml_orphan_ephemerons(domain_state);
     caml_orphan_finalisers(domain_state);
 
+    if (domain_state->marking_done &&
+        domain_state->sweeping_done) {
path: runtime/domain.c
user: login: sadiqj
body: You are right, I missed that `ephe_mark` could reset the marking flag and the potential cycle from orphaning finalisers. This makes the conditions a lot clearer.

Thanks, I think I'm happy with this. Let's wait to see whether @jmid can break it.
created_at: 2025-05-19T10:54:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2118,19 +2136,24 @@ void caml_domain_terminate(bool last)
     caml_plat_unlock(&all_domains_lock);
   }
 
-  /* domain_state may be re-used by a fresh domain here (now that we
-   * have done remove_from_stw_domains and released the
-   * all_domains_lock). However, domain_create() won't touch it until
-   * it has claimed the domain_lock, so we hang onto that while we are
-   * tearing down the state. */
+  CAMLassert(last ||
+             (domain_state->marking_done
+              && domain_state->sweeping_done));
path: runtime/domain.c
user: login: jmid
body: This should probably also just use the new helper function `marking_and_sweeping_done(domain_state)` to avoid duplicating the condition.
created_at: 2025-05-19T14:49:39Z
author_association: MEMBER
diff_hunk: @@ -2118,19 +2136,24 @@ void caml_domain_terminate(bool last)
     caml_plat_unlock(&all_domains_lock);
   }
 
-  /* domain_state may be re-used by a fresh domain here (now that we
-   * have done remove_from_stw_domains and released the
-   * all_domains_lock). However, domain_create() won't touch it until
-   * it has claimed the domain_lock, so we hang onto that while we are
-   * tearing down the state. */
+  CAMLassert(last ||
+             (domain_state->marking_done
+              && domain_state->sweeping_done));
path: runtime/domain.c
user: login: gasche
body: Done, thanks!

For the record, I thought of adding a similar assertion after the `if (last)` case below -- because that case supposedly restores the invariant in the `(last)` case that were established in the other case by the main loop. But I don't think it actually holds: what we need to call `caml_free_shared_heap` is that there are no pools left on the current domain (in particular no unswept pools), but this does not necessarily entail that the `->sweeping_done` boolean has been set.

Now that I write this comment, I see that I probably need a different assertion `caml_shared_heap_is_empty(caml_domain_state*)` to clarify the nuance. I'll try to add that.
created_at: 2025-05-19T15:06:06Z
author_association: MEMBER
diff_hunk: @@ -2118,19 +2136,24 @@ void caml_domain_terminate(bool last)
     caml_plat_unlock(&all_domains_lock);
   }
 
-  /* domain_state may be re-used by a fresh domain here (now that we
-   * have done remove_from_stw_domains and released the
-   * all_domains_lock). However, domain_create() won't touch it until
-   * it has claimed the domain_lock, so we hang onto that while we are
-   * tearing down the state. */
+  CAMLassert(last ||
+             (domain_state->marking_done
+              && domain_state->sweeping_done));
path: runtime/domain.c
user: login: gasche
body: Done, in a new commit.
created_at: 2025-05-19T15:15:24Z
author_association: MEMBER

PR comments:
user: login: gasche
created_at: 2025-05-11T07:58:30Z
body: Several parts of the domain-teardown code (after exiting the STW participant set) combine two steps together: "(1) orphan some domain-local runtime structure into a global adoption structure, (2) free the domain-local structure" -- `caml_teardown_shared_heap` on trunk, but also `caml_memprof_delete_domain` for example. In the present PR I split this logic in two for `caml_teardown_shared_heap`, with (1) happening before we exit the STW set, and only (2) happening afterwards.

If we decide to go this route, I think that we should probably apply the same split to other teardown functions for consistency, such as the memprof stuff. This may or may not be needed for correctness for each structure, but it gives the convenient invariant that all domain structures of this kind are either owned by an active domain or listed for global adoption.
author_association: MEMBER
user: login: ghost
created_at: 2025-05-11T11:11:55Z
body: You should probably update the comment on line 2152 to refer to `caml_orphan_shared_heap` now.
author_association: NONE
user: login: gasche
created_at: 2025-05-11T12:13:54Z
body: Note to self: with the patchset as currently proposed, the following code that is after the STW exit becomes a no-op and this may be a bug:

```c
  if (last)
    caml_finalise_heap();
```

This code is supposed to call C finalisers attached to custom objects in on the major heap -- only when it is the last domain that is terminating. It becomes a no-op because it only traverses pools of the current domain shared heap, and with this PR this set will be empty, they have all been orphaned at this point.

The code could be fixed by changing `caml_finalise_heap` to finalise the pools of the global freelist, instead of the pools of the current domain.

However, I'm not sure that this matters, because the code only traverses *unswept* pools, and I don't know why there would be any remaining at this point of the domain-termination logic. (I hope there is none, because I added a `CAMLassert` assertion to this effect in `caml_free_shared_heap` which is called around there.) It may be the case that this code is already a no-op; again, a GC expert would help.
author_association: MEMBER
user: login: jmid
created_at: 2025-05-12T11:10:53Z
body: Thanks for the summary and the PR!

In the CI I see core dumps and assertion failures `runtime/shared_heap.c; line 163 ### Assertion failed: !heap->unswept_avail_pools[i]` indicating that sweeping is not done when attempting to orphan the shared heap.
Furthermore, the `caml_orphan_shared_heap` is enclosed in a `while`-loop, so it may potentially be run multiple times, which I'm a bit uneasy with.

I've experimented with moving the orphaning call to the final loop iteration:
```diff
diff --git a/runtime/domain.c b/runtime/domain.c
index 5521783551..e2d69e88ea 100644
--- a/runtime/domain.c
+++ b/runtime/domain.c
@@ -2072,7 +2072,6 @@ void caml_domain_terminate(bool last)
 
     caml_orphan_ephemerons(domain_state);
     caml_orphan_finalisers(domain_state);
-    caml_orphan_shared_heap(domain_state->shared_heap);
 
     /* No need to check for interrupts if we are the last domain running. */
     if (last) {
@@ -2102,6 +2101,8 @@ void caml_domain_terminate(bool last)
       s->terminating = 0;
       s->running = 0;
 
+      caml_orphan_shared_heap(domain_state->shared_heap);
+
       /* Remove this domain from stw_domains */
       remove_from_stw_domains(domain_self);
```
With this patch both the reproducer above passes, as do the testsuite/parallel and testsuite/compaction tests.
There are however a number of other supposedly unrelated dynlink and debugger test failures...
The downside of the above patch is that the orphaning moves into the `all_domains_lock`ed region,
which could potentially slow termination down a bit...
author_association: MEMBER
user: login: jmid
created_at: 2025-05-12T13:43:34Z
body: OK, after looking some more at `caml_orphan_shared_heap` I'm less uneasy with doing duplicate calls.
However, there's still a precondition of having finished marking and sweeping.
Commit 6a6602776e03026d9fdc5f116efea39cb8aaf310 from the rebased [orphan-shared-heap-before-STW-exit-take2](https://github.com/jmid/ocaml/tree/orphan-shared-heap-before-STW-exit-take2) branch thus adds a guard with these requirements and moves the call outside the `all_domains_locked` region.

The result passes the relevant tests, however a number of unrelated ones fail under a musl-configured compiler:
```
$ ./configure --disable-ocamldoc "CC=musl-gcc" "CFLAGS=-Os" "ASPP=musl-gcc -c"
$ make -j8
$ cd testsuite
$ make parallel
[...]

List of failed tests:
    tests/lib-dynlink-bytecode/main.ml
    tests/lib-dynlink-domains/main.ml
    tests/lib-dynlink-init-info/test.ml
    tests/lib-dynlink-initializers/test10_main.ml
    tests/lib-dynlink-initializers/test1_main.ml
    tests/lib-dynlink-initializers/test2_main.ml
    tests/lib-dynlink-initializers/test3_main.ml
    tests/lib-dynlink-initializers/test5_main.ml
    tests/lib-dynlink-initializers/test6_main.ml
    tests/lib-dynlink-initializers/test7_main.ml
    tests/lib-dynlink-initializers/test8_main.ml
    tests/lib-dynlink-initializers/test9_main.ml
    tests/lib-dynlink-packed/loader.ml
    tests/lib-dynlink-pr4229/main.ml
    tests/lib-dynlink-pr4839/test.ml
    tests/lib-dynlink-pr6950/loader.ml
    tests/lib-dynlink-pr9209/dyn.ml
    tests/lib-dynlink-private/test.ml
    tests/native-debugger/linux-gdb-amd64.ml
    tests/tool-debugger/basic/debuggee.ml
    tests/tool-debugger/dynlink/host.ml
    tests/tool-debugger/find-artifacts/debuggee.ml
    tests/tool-debugger/module_named_main/main.ml
    tests/tool-debugger/no_debug_event/noev.ml
    tests/tool-debugger/printer/debuggee.ml

Summary:
  1401 tests passed
    81 tests skipped
    25 tests failed
     0 tests not started (parent test skipped or failed)
     0 unexpected errors
  1507 tests considered
#### Something failed. Exiting with error status.
```
I've thus checked that this is the case for a fresh `trunk` build with the above `musl` configuration, and hence unrelated to the proposed patch.
author_association: MEMBER
user: login: gasche
created_at: 2025-05-12T14:06:05Z
body: Conditioning on `marking_done` and `sweeping_done` is better, thanks! But I would keep the code above the `if (last)` guard if possible, it makes things easier to reason about. The intent is to run in a loop indeed: if after this orphaning there is an STW request that comes in, we must handle it and then retry.
author_association: MEMBER
user: login: sadiqj
created_at: 2025-05-12T14:18:21Z
body: I plan to look at this later in the week but I think it's also worth cc'ing @stedolan and @NickBarnes 

The compactor issue affects the modified compactor on flambda-backend and I suspect there will be interactions between this and markdelay.
author_association: CONTRIBUTOR
user: login: gasche
created_at: 2025-05-12T23:35:20Z
body: @jmid I updated the PR to take your comments and mine into account. Whenever you have time, a new round of testing would be warmly welcome.
author_association: MEMBER
user: login: jmid
created_at: 2025-05-13T11:45:04Z
body: I've given this a round of `multicoretest`ing across platforms and configs and it holds up well! :+1: 

As mentioned in https://github.com/ocaml/ocaml/issues/13739#issuecomment-2876097445 I also confirmed our hypothesis of elements being added to `pool_freelist` in the window after having checked the precondition up front. As such, I think it would make sense to reconfirm the assumption, as it is central to the correctness of compaction. In my [orphan-shared-heap-before-STW-exit](https://github.com/jmid/ocaml/tree/orphan-shared-heap-before-STW-exit) branch commit 26e9f556be7c8d34bd66332b142e351c255b4d0c I thus add this assertion, just before updating the major heap pointers.

Realizing that the existing `testsuite/tests/compaction/test_compact_manydomains.ml` tests compaction with "domain-local heap-allocated data" only, I also suggest adding a cut down version of the test case from #13739 in commit c6cbad79aaa0f0caabd41bce969751671508fc3a. With changes in the pipeline regarding Gc-pacing, stronger Gc tests will not hurt. :slightly_smiling_face: 
author_association: MEMBER
user: login: gasche
created_at: 2025-05-13T14:22:52Z
body: @jmid I can't include your test directly as written: I hear from people more familiar with the testsuite workings that we are resource-constrained on some of our test machines, so we need to reduce the amount of parallelism in CI environments and possibly make the number of rounds configurable. I don't know exactly what should be done in this case, whereas I'm happy with the changes already committed, so I would like this to stay as a separate PR to be reviewed independently. I would propose that you submit it, but I can also do it if you prefer.
author_association: MEMBER
user: login: gasche
created_at: 2025-05-13T14:30:49Z
body: Status report:
- Compaction in presence of concurrent domain termination is currently broken and will result in a invalid heap in some interleavings (found by @jmid via randomized testing, but could occur in the wild). The PR is designed to fix this issue.
- I'm happy with the current state of the PR and believe it is ready for reviewing and merging.
- @jmid has tested it, which confirms that it appears to fix the reported bug, and also that it does not introduce any obvious bug at the same time.

Could we get someone to review it? I would think of @damiendoligez, @stedolan and @NickBarnes, but maybe @OlivierNicole or @gadmm.
author_association: MEMBER
user: login: gasche
created_at: 2025-05-13T14:43:25Z
body: Thinking out loud: I mentioned that it would be nice to change `memprof` to follow the same logic here (orphan before leaving the STW set). I see what changes should be done and I'm happy to do it, but I would rather do it in a follow-up PR. (One reason is that each change asks Jan to do yet another round of testing, and I would rather minimize the iterations to save his time.)
author_association: MEMBER
user: login: jmid
created_at: 2025-05-19T19:28:23Z
body: Thanks for the revision! I very much appreciate the additional comments you've added :+1: 
Both the fix and the split orphaning depending on being `last` or not LGTM. 

> Let's wait to see whether @jmid can break it.

I've given this a spin on the various CI configurations [here](https://github.com/ocaml-multicore/multicoretests/tree/refs/heads/try-gasche-orphan-fix) and also built the compiler locally with `musl` and tested the crashing example from above repeatedly. In all cases this has held up nicely! :smiley:
author_association: MEMBER
user: login: gasche
created_at: 2025-05-20T10:43:44Z
body: Great! Thanks both, for the review and the tests. Merging.

(This is a somewhat invasive to the `domain_terminate` implementation. I decided to keep the "review" commits, which are more invasive than the first iteration, separate, in case later we find out that a regression was introduced specifically in this part -- or maybe we want to cherry-pick the less-diffy first few commits, etc.)
author_association: MEMBER