Only drop vertices when dropping a graph if vertices aren't used
elsewhere