Hoops/bdwgc — pull request #1 by Hoops (2013-12-21 13:47:31+00:00)
Title: Try to stop excessive mark stack growth in parallel mode.

Following on from my previous patch, this addresses an issue where the mark stack grows excessively during parallel mark when building some modules in Guile (http://www.gnu.org/software/guile/), which uses custom kinds and mark procedures. These implement things like weak hash tables and iterate over the members marking entries using the supplied macro `GC_MARK_AND_PUSH`.

When the mark stack limit is hit in `PUSH_OBJ`, from the depths of the `GC_MARK_AND_PUSH` macro, it will call `GC_signal_mark_stack_overflow` . This function sets `GC_mark_stack_too_small` to `TRUE`, which seems good for the single-threaded case as we are talking about the global mark stack. But in parallel mark the overflow is caused by the local mark stack overflowing. So my thought is that this flag should not be set in this case. When the local mark stack is returned in `GC_return_mark_stack`, that function will check for overflow of the global stack and set `GC_mark_stack_too_small` accordingly.

Without this patch, I have seen the mark stack grow to 2GB for a heap of only 60mb! It is doubled each time that `GC_mark_stack_too_small` is set during a single collection, so can get quite large very quickly. For the same application with this patch, the global mark stack grows in the same way as the non-parallel mode,
i.e. it is only resized a couple of times.

I'm still getting used to this code base, so it would be good to hear your thoughts on this.