[core] Really remove build directory