ohler55/oj — pull request #116 by skaes (2013-11-17 10:32:46+00:00)
Title: don't try to mark out of scope value stacks

Hi Peter,

I have found the root cause for the segfaults I have been experiencing. This patch fixes the problem.

The problem lies with with mark function for value stacks.

When Oj.load returns, it will have created a T_DATA object on Ruby's heap and a reference to it on the C-stack. The T_DATA object then contains a pointer to a memory segment on the C-stack (a _ValStack) which is no longer in scope. Now assume some other C function is called, which reserves stack space, but doesn't immediately overwrite it. If at this point GC kicks in, and finds the reference to the original _ValStack, it will call the corresponding mark function. If the _ValStack memory region has been modified, it will likely cause a segfault.

The patch fixes the problem by clearing the data pointer inside the T_DATA object.

Note that simply overwriting the pointer to the T_DATA object on the stack is not good enough, as the mark phase could find the T_DATA simply because some other part of the program has put the integer value of the T_DATA reference into some memory location by pure chance. This is why Ruby's GC is called conservative: it will sometimes collect stuff which is no longer accessible.