make flag checking more efficient and prevent an Int3() on bad data

git-svn-id: svn://svn.icculus.org/fs2open/trunk/fs2_open@11109 387891d4-d844-0410-90c0-e4c51a9137d3