Fixes a check for what mmap returns.

On error mmap returns MAP_FAILED(-1) not null.
FreeBSD was checking the return correctly, Linux was not.
This was noticed by triad attempting to run Dolphin under valgrind and not getting a memory space under the 2GB limit(Because -1 wraps around on
unsigned obviously)