Try to handle BUILD_LIBRARIES_ONLY correctly

Also, try to fix OS X build