rt: Set the stack depth limit to 1GB. Abort on error.

People hit the recursion depth limit too often, it's not possible
to unwind reliably from out-of-stack.

Issues #3555, #3695