Fix an integer overflow in isolate

When using a high memory limit (2GB), an integer overflow can happen in the sandbox. This commit should fix that (in the way it was already done for the stack limit).