Free the endpoints of an edge when we free the startpoint