build system: add flag for mex compilation on OS X 10.9

(cherry picked from commit cf910cb64f6e0468de123fe9fdfe4fc6317774b4)