Fix up ghost_by_vertex to some extent