Fix #2957 - FreeBSD RAND_MAX is 0x7ffffffd, not 0x7fffffff, so we can't always assume that it's the latter.  Also move the check earlier in the code, if it's going to break at all, break immediately on any non-conforming platform so we can fine tune this some more.

git-svn-id: svn://svn.icculus.org/fs2open/trunk/fs2_open@10129 387891d4-d844-0410-90c0-e4c51a9137d3