fix interaction of continuations and stack overflow

There was an off-by-one error in trimming overflow
records in a captured continuation.

I provoked the crash by running the program below on Mac OS X;
resizing the frame caused a crash. It has something to do with the
`try-atomic' implementation, I think. I wasn't able to make a test
case in a half-hour of trying, however, and I'm giving up for now.

(define f (new frame% [label "deep"]))
(define b (new button%
               [parent f]
               [label "0"]))
(send f show #t)
(let loop ([n 0] [m 0])
  (if (= n 10000)
      (begin
        (send b set-label (format "~a" m))
        (for ([i 10]) (yield))
        (loop 0 (add1 m)))
      (cons 1 (loop (add1 n) m))))
increase stack safety margin for 64-bit platforms

Make the stack-safety margin twice as big for 64-bit platforms
as 32-bit platforms. That was already done for Windows, but it's
also needed for Mac OS X. Also, double-check that there's a
good amount of space on the stack before calling a foreign
function.