Remove the now ignored directory builds