Remove ignores for irrelevent directories