Fixes a check for what mmap returns.

On error mmap returns MAP_FAILED(-1) not null.
FreeBSD was checking the return correctly, Linux was not.
This was noticed by triad attempting to run Dolphin under valgrind and not getting a memory space under the 2GB limit(Because -1 wraps around on
unsigned obviously)
Merge pull request #705 from Sonicadvance1/fix-memoryutil-check

Fixes a check for what mmap returns.