Make sure library flags are set for only OSX