Add BUILD_EXTRA_LIBS flag

That way we don't build libraries that aren't needed yet.