Fix prompt coloring when traversing non-editable directories