Don't rebuild if no flag is present

It happens after you clean, and there is no need to clean
again in that case.