Remove the ignored directory