Do a proper group check when closing contact. Fix #449