Prevent LLVM from rebuilding every time

Only happens when assertions are enabled